Data interaction methods, devices, equipment, and media based on wearable devices

Abstract

This disclosure relates to a data interaction method, apparatus, device, and medium based on wearable devices. The method includes: identifying a terminal device within a defined spatial range using the wearable device; generating a virtual wireframe and network protocol address for the terminal device; recognizing a user's palm within the defined spatial range to generate a virtual hand corresponding to the user's palm in the wearable device; determining a data interaction flow between the wearable device and the terminal device when the distance between the virtual hand and the virtual wireframe is less than a distance threshold and the target gesture of the virtual hand matches a preset interactive gesture; and performing data interaction between the wearable device and the terminal device according to the data interaction flow and the network protocol address. This method utilizes wearable devices to achieve data interaction between devices through gesture interaction, resulting in a more realistic interactive experience and more convenient operation.

Description

Technical Field

[0001] This disclosure relates to the field of mixed reality technology, and in particular to a data interaction method, apparatus, device and medium based on wearable devices. Background Technology

[0002] In related technologies, data interaction between mobile devices mainly involves data transmission through wired or wireless connections. However, this is limited by the connection lines between devices, which can easily lead to data transmission obstruction. Furthermore, the data interaction process requires establishing communication connections between devices, making the process cumbersome and affecting efficiency. For example, during an indoor meeting, when a user needs to present a document on a projector, they must first establish a communication connection between the mobile device and the projector via wired or wireless means before transferring the data file from the mobile device to the projector. If the connection between the mobile device and the projector is obstructed, or if the connection is interrupted during transmission, normal data interaction between the mobile device and the projector will be impossible. Summary of the Invention

[0003] To overcome the problems existing in related technologies, this disclosure provides a data interaction method, apparatus, device and medium based on wearable devices.

[0004] According to a first aspect of the present disclosure, a data interaction method based on a wearable device is provided, comprising:

[0005] Based on the wearable device, the terminal devices within a set spatial range are identified, and a virtual wireframe and network protocol address of the terminal devices are generated;

[0006] In response to detecting a user's palm within the defined spatial range, the user's palm is identified to generate a virtual hand corresponding to the user's palm in the wearable device;

[0007] If the distance between the virtual hand and the virtual wireframe is less than a distance threshold, and the target gesture of the virtual hand matches a preset interactive gesture, the data interaction process between the wearable device and the terminal device is determined.

[0008] Data interaction is performed between the wearable device and the terminal device according to the data interaction process and the network protocol address.

[0009] Optionally, determining the data interaction process between the wearable device and the terminal device includes:

[0010] Monitor the dynamic trajectory of the virtual hand to generate the gestures of the virtual hand;

[0011] When the gesture action is a set extraction action, the current extraction target of the virtual hand in the wearable device is determined;

[0012] If the current extraction target is not within the display range of the virtual wireframe, then the data interaction process is determined to be a terminal device display process;

[0013] If the current extraction target is within the display range of the virtual wireframe, then the data interaction process is determined to be a wearable device display process.

[0014] Optionally, the step of performing data interaction between the wearable device and the terminal device according to the data interaction process and the network protocol address includes:

[0015] When the data interaction process is the wearable device display process, the network protocol address is sent to the network end. The network protocol address is used to instruct the network end to request the first target data corresponding to the current extraction target from the terminal device according to the network protocol address.

[0016] In response to receiving the first target data fed back from the network, the first target data is displayed at a set location on the wearable device;

[0017] When the data interaction process is the terminal device display process, the second target data corresponding to the current extraction target and the network protocol address are sent to the network end. The network protocol address is used to instruct the network end to send the second target data to the terminal device according to the network protocol address, so as to display the second target data in the terminal device.

[0018] Optionally, the step of recognizing the user's palm to generate a virtual hand corresponding to the user's palm in the wearable device includes:

[0019] Key point recognition is performed on the user's palm to generate multiple palm key points corresponding to the user's palm;

[0020] Based on the user's palm displayed in the wearable device, the multiple key points of the palm are connected to generate the virtual hand.

[0021] Optionally, the method includes:

[0022] The movement trajectory of the multiple key points of the palm in the virtual hand is monitored to generate the target gesture of the virtual hand;

[0023] If the similarity between the target gesture and the preset interactive gesture reaches a similarity threshold, the target gesture is determined to match the preset interactive gesture.

[0024] Optionally, the step of identifying terminal devices within a defined spatial range based on the wearable device and generating a virtual wireframe and network protocol address for the terminal devices includes:

[0025] The wearable device is used to identify the terminal device within a set spatial range to generate the appearance features of the terminal device.

[0026] The virtual wireframe is generated based on the described appearance features;

[0027] The virtual wireframe is sent to the network end, and the virtual wireframe is used to instruct the network end to determine the network protocol address of the terminal device based on the virtual wireframe.

[0028] Receive the network protocol address fed back by the network terminal.

[0029] Optionally, the step of identifying terminal devices within a defined spatial range based on the wearable device and generating the network protocol address of the terminal devices includes:

[0030] The wearable device identifies terminal devices within a defined spatial range to determine the actual spatial distance between the wearable device and the terminal devices.

[0031] The location information and actual spatial distance of the wearable device are reported to the network terminal. The location information and actual spatial distance are used to instruct the network terminal to determine the network protocol address of the terminal device based on the location information and actual spatial distance.

[0032] Receive the network protocol address fed back by the network terminal.

[0033] According to a second aspect of the present disclosure, a data interaction device based on a wearable device is provided, comprising:

[0034] The first generation module is configured to identify terminal devices within a set spatial range based on the wearable device, and generate virtual wireframes and network protocol addresses for the terminal devices.

[0035] The second generation module is configured to, in response to detecting a user's palm within the set spatial range, identify the user's palm to generate a virtual hand corresponding to the user's palm in the wearable device;

[0036] The determination module is configured to determine the data interaction process between the wearable device and the terminal device when the distance between the virtual hand and the virtual wireframe is less than a distance threshold and the target gesture of the virtual hand matches a preset interactive gesture.

[0037] The execution module is configured to perform data interaction between the wearable device and the terminal device according to the data interaction process and the network protocol address.

[0038] According to a third aspect of the present disclosure, a wearable device is provided, comprising:

[0039] processor;

[0040] Memory used to store processor-executable instructions;

[0041] The processor is configured to implement the steps of any of the methods described in the first aspect of this disclosure when executing the executable instructions.

[0042] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, having stored thereon computer program instructions that, when executed by a processor, implement the steps of the data interaction method based on a wearable device provided in the first aspect of the present disclosure.

[0043] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0044] The above method identifies terminal devices within a defined spatial range using wearable devices, generating virtual wireframes and network protocol addresses for these devices. In response to the detection of a user's palm within the defined spatial range, the wearable device identifies the user's palm and generates a virtual hand corresponding to it. If the distance between the virtual hand and the virtual wireframe is less than a distance threshold, and the target gesture of the virtual hand matches a preset interactive gesture, the data interaction flow between the wearable device and the terminal device is determined. Based on the data interaction flow and the network protocol address, data interaction occurs between the wearable device and the terminal device. Thus, by using wearable devices for gesture interaction, data interaction between devices is achieved, resulting in a more realistic interactive experience and more convenient operation.

[0045] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0046] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0047] Figure 1 This is a flowchart illustrating a data interaction method based on a wearable device according to an exemplary embodiment.

[0048] Figure 2This is an example diagram illustrating a data interaction method for mixed reality glasses according to an exemplary embodiment.

[0049] Figure 3 This is a flowchart illustrating a method for generating a mixed reality scene according to an exemplary embodiment.

[0050] Figure 4 This is a flowchart illustrating a method for determining a gesture event according to an exemplary embodiment.

[0051] Figure 5 This is a flowchart illustrating a data interaction method based on a wearable device according to an exemplary embodiment.

[0052] Figure 6 This is a schematic diagram illustrating a virtual hand according to an exemplary embodiment.

[0053] Figure 7 This is a flowchart illustrating a target gesture recognition method according to an exemplary embodiment.

[0054] Figure 8 This is a schematic diagram illustrating a wearable device demonstration process according to an exemplary embodiment.

[0055] Figure 9 This is a schematic diagram illustrating a terminal device display process according to an exemplary embodiment.

[0056] Figure 10 This is a flowchart illustrating a data interaction method based on mixed reality glasses according to an exemplary embodiment.

[0057] Figure 11 This is a block diagram illustrating a data interaction device based on a wearable device according to an exemplary embodiment.

[0058] Figure 12 This is a block diagram illustrating a wearable device according to an exemplary embodiment. Detailed Implementation

[0059] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0060] It should be noted that all actions involving the acquisition of signals, information, or data in this application are carried out in compliance with the relevant data protection laws and policies of the country where the application is located, and with the authorization granted by the owner of the relevant device.

[0061] Before describing the data interaction method in this application, it is necessary to introduce the wearable device used in this application. The wearable device in this application is used to implement mixed reality technology. Mixed reality (MR) scenes are a form between virtual and real scenes, including augmented reality scenes and augmented virtual scenes. It refers to a new visual environment created by merging reality and virtuality. In this new visual environment, physical and virtual objects coexist, and various interactions can occur between them. The information flow interaction between multiple devices in this application uses a wearable device equipped with mixed reality technology as the main body, and achieves indirect communication between devices by embedding services in multiple devices.

[0062] In related technologies, virtual objects have been endowed with more realistic attributes in various ways. For example, virtual objects are fixed using spatial positioning technology; clear hierarchical distinctions are achieved between real and virtual objects through virtual-real occlusion technology; and virtual objects are synchronized with the object's lighting information through lighting calculation technology. Immersive interaction in mixed reality allows virtual objects to integrate into every corner of life, shifting the transmission medium from mobile phone screens to thinner and more portable wearable devices. Image recognition and data transmission technologies are used to acquire specific information displayed on other electronic devices, and more data interactions are achieved in the mixed reality space of wearable devices based on corresponding control gestures.

[0063] Figure 1 This is a flowchart illustrating a data interaction method based on a wearable device according to an exemplary embodiment, such as... Figure 1 As shown, this method is used in wearable devices and includes the following steps.

[0064] In step S101, the wearable device identifies the terminal devices within a set spatial range and generates a virtual wireframe and network protocol address for the terminal devices.

[0065] It is worth mentioning that this embodiment is applied to wearable devices equipped with mixed reality technology. These wearable devices can be mixed reality glasses, mixed reality helmets, etc., and display a mixed reality scene on the wearable device's display screen. When a user wears the wearable device, the mixed reality scene is presented to the user through the display screen. The spatial range displayed to the user within the mixed reality scene by the display screen is the set spatial range that the wearable device can capture. To facilitate mixed reality interaction, after the user wears the wearable device, the wearable device scans the set spatial range to identify terminal devices within that range. For example, terminal devices typically have relatively fixed appearance features, such as televisions, mobile phones, and personal computers, all of which have a quadrilateral frame and a display device. The device can identify terminal devices within the set spatial range based on these appearance features and generate a virtual wireframe of the terminal device at the corresponding position on the wearable device's display screen based on the terminal device's frame.

[0066] It should be noted that in this embodiment, the principle of data interaction between the wearable device and terminal devices within a defined spatial range is to share the data currently displayed on the terminal device's screen to the local area network (LAN) or the internet, and then send this data to the wearable device for display via the LAN or the internet. Alternatively, the data currently displayed on the wearable device can be shared to the LAN or the internet, and then sent to the terminal device for display via the LAN or the internet. This data can be image data, video data, file data, etc., which are not limited in this embodiment. Therefore, the corresponding terminal device is a smart device that can access the network. This smart device can include: LAN devices that need to connect to the internet through a relay router, such as televisions, laptops, and tablets, and internet devices that can directly connect to the internet through a network card, such as mobile phones and desktop computers connected via network cables. For example... Figure 2 This is an example diagram illustrating a data interaction method for mixed reality glasses according to an exemplary embodiment, such as... Figure 2 As shown, in this embodiment, the wearable device is a mixed reality glasses. When the terminal device is a local area network (LAN) device (TV, personal computer), the mixed reality glasses and the LAN device are connected to the Internet through the same router. In this case, the router acts as a data relay interface to realize indirect data interaction between the mixed reality glasses and the LAN device. When the terminal device is an Internet device (smartphone), the mixed reality glasses have built-in Internet connectivity and can access the Internet. In this case, the Internet acts as a data relay interface between the mixed reality glasses and the Internet device to realize indirect data interaction between the mixed reality glasses and the Internet device.

[0067] A network protocol address serves as a unique identifier for a terminal device during network data interaction. Different terminal devices have different network protocol addresses. By identifying the network protocol address, the identity of the terminal device during network data interaction can be determined, thereby identifying the data file currently being viewed by the terminal device. For example, in this embodiment, for local area network (LAN) devices, the corresponding router relay station in the LAN has a pre-configured mapping relationship between the appearance features of each terminal device and its network protocol address. When a user wears the wearable device, it identifies terminal devices within a defined spatial range, extracting the appearance features of the corresponding terminal devices. These appearance features may include size, color, brand, etc., and are sent to the router relay station. The router relay station determines the network protocol address of the terminal device based on the appearance features and the mapping relationship, and then feeds this network protocol address back to the wearable device. The wearable device displays this network protocol address at the corresponding position in the virtual wireframe of the terminal device. For example... Figure 3 This is a flowchart illustrating a method for generating a mixed reality scene according to an exemplary embodiment, such as... Figure 3 As shown, the method includes the following steps.

[0068] (1) Scan the scene within a set range using wearable devices to generate map information;

[0069] (2) Identify the terminal devices in the scene based on the map information;

[0070] (3) If there is a terminal device that has been identified in the scene, the virtual wireframe corresponding to the terminal device is retrieved from the database according to the device name of the terminal device, and the virtual wireframe is generated according to the display position of the terminal device in the wearable device; if the terminal device in the scene has not been identified by the wearable device, the corresponding virtual wireframe is generated based on the position of the terminal device in the scene.

[0071] (4) Identify terminal devices: Enter the network IP address corresponding to the terminal device into the wearable device to generate map files for the corresponding scene.

[0072] Optionally, the above-described step of generating the network protocol address of the terminal device may include:

[0073] The wearable device identifies terminal devices within a defined spatial range to determine the actual spatial distance between the wearable device and the terminal device.

[0074] The location information and actual spatial distance of the wearable device are reported to the network end. The location information and actual spatial distance are used to instruct the network end to determine the network protocol address of the terminal device based on the location information and actual spatial distance.

[0075] The network protocol address received from the network end.

[0076] It is worth mentioning that in this embodiment, the wearable device and the terminal device are connected to the same network. For example, if the terminal device is a local area network (LAN) device, the wearable device and the terminal device are connected to the Internet through the same router relay station; if the terminal device is an Internet device, the wearable device and the terminal device are connected to the Internet through the same base station. For example, in this embodiment, the wearable device has precise positioning and distance detection functions. The actual spatial distance between the wearable device and the terminal device is determined by the display ratio and display distance of the terminal device on the display device of the wearable device. The actual spatial distance can be a relative three-dimensional coordinate in a three-dimensional coordinate system. The actual spatial distance and the location information of the wearable device are sent to the network end. The network end determines the location of the wearable device based on the location information and, according to the precise positioning method, determines multiple other terminal devices within a preset range of the wearable device. Based on the actual spatial distance between the wearable device and the terminal device, the network end determines the terminal device displayed on the display device of the wearable device from among the multiple other terminal devices. Based on the data interaction records between the terminal device and the network end, the network protocol address corresponding to the terminal device is determined and fed back to the wearable device.

[0077] It should be noted that in this embodiment, the wearable device may scan and determine multiple terminal devices within a set space range. The virtual wireframes and network protocol addresses corresponding to multiple terminal devices can be determined in the manner described above, thereby increasing the practicality of the wearable device and enabling it to better achieve data interaction with any terminal device within the set space range.

[0078] Alternatively, in another embodiment, the step of generating the network protocol address of the terminal device may further include:

[0079] Based on wearable devices, terminal devices within a defined spatial range are identified to generate the appearance features of the terminal devices;

[0080] Based on the appearance features, generate a virtual wireframe;

[0081] The virtual wireframe is sent to the network end, and the virtual wireframe is used to instruct the network end to determine the network protocol address of the terminal device based on the virtual wireframe.

[0082] The network protocol address received from the network end.

[0083] It is worth mentioning that in this embodiment, there is a unique correspondence between the virtual wireframes of each terminal device within the defined spatial range and the terminal device itself. A mapping relationship is established on the network side between the virtual wireframes and the corresponding network protocol addresses of the terminal devices. By consulting this mapping relationship, the network protocol address corresponding to the virtual wireframe can be determined. For example, in this embodiment, the wearable device identifies the terminal devices within the defined range, generates the appearance features of the terminal devices, and generates virtual wireframes based on these appearance features. The virtual wireframe is sent to the network side, which determines the network protocol address corresponding to the virtual wireframe based on the aforementioned mapping relationship and feeds back the network protocol address of the terminal device to the wearable device.

[0084] In step S102, in response to detecting the user's palm within a set spatial range, the user's palm is identified to generate a virtual hand corresponding to the user's palm in the wearable device.

[0085] It is worth mentioning that in this embodiment, data interaction and display in the wearable device are mainly controlled through user gestures. By detecting the user's hand gestures on the wearable device's display, video playback, image display, file display, and data interaction between the wearable device and the terminal device are controlled. For example, a fixed gesture can be used to control the display of corresponding image data on the wearable device's display. Because the wearable device is equipped with mixed reality technology, the image data is displayed on the real-world scene captured by the wearable device within a set spatial range. The user can also control the display position of the image data within that real-world scene using corresponding gestures. Therefore, to identify the user's intent, the wearable device detects the user's hand within a set spatial range. It should be noted that to prevent misidentification of the user's hand within the set spatial range, typically, the detection is centered on the user and detects hand images within a preset threshold range. When a hand image is detected within this preset threshold range, it is determined that the hand image is the user's hand, and the hand is identified to determine the corresponding gesture.

[0086] For example, in this embodiment, important features such as finger size, palm size, and finger position of the user's hand can be detected, and a virtual hand representing the user's hand can be generated in the wearable device based on these features. It is worth noting that changes in the user's hand within the wearable device include positional changes and gesture changes. When it is necessary to drag virtual data displayed on the wearable device, the user's hand needs to drag the virtual data, resulting in a positional change. When it is necessary to operate the virtual data displayed on the wearable device, the user's fingers corresponding to the palm need to issue operation commands to the wearable device, resulting in a gesture change. Therefore, the virtual hand includes the user's corresponding palm and fingers to represent the positional and gesture change characteristics of the user's hand. The display position of the virtual hand in the wearable device corresponds to the display position of the user's palm. When the user's palm changes within the wearable device, the virtual hand will follow the user's palm, changing or moving within the display device of the wearable device.

[0087] In step S103, when the distance between the virtual hand and the virtual wireframe is less than a distance threshold and the target gesture of the virtual hand matches the preset interactive gesture, the data interaction process between the wearable device and the terminal device is determined.

[0088] For example, after generating a virtual hand through the above steps, the distance between the virtual hand and the virtual wireframe is detected to determine whether relevant data interaction has occurred between the user's palm and the terminal device. When the user's palm touches the terminal device on the wearable device's display, and the corresponding virtual hand intersects with the virtual wireframe, it is determined that the user intends to interact with the terminal device. Then, the target gesture corresponding to the user's virtual hand is detected. When the target gesture matches a preset interaction gesture, the data interaction process between the wearable device and the terminal device is determined through the target gesture.

[0089] For example, the data interaction process between the wearable device and the terminal device includes (1) extracting the currently displayed data from the terminal device to the wearable device for data display through a relay station; and (2) extracting the currently displayed data from the wearable device to the terminal device for display through a relay station. Therefore, the preset interaction gestures between the wearable device and the terminal device in this embodiment include: data grabbing gestures and data releasing gestures. The data interaction process between the wearable device and the terminal device can be determined by detecting the position of the virtual hand corresponding to the gesture. Figure 4 This is a flowchart illustrating a method for determining a gesture event according to an exemplary embodiment, such as... Figure 4 As shown, the determination method includes the following steps.

[0090] (1) Recognize the position and gesture type of the virtual hand;

[0091] (2) Project the corresponding sensing rays directly in front of the virtual hand;

[0092] (3) When the sensing ray hits the virtual frame corresponding to the terminal device, determine the distance between the virtual hand and the virtual frame;

[0093] (4) When the distance is less than the threshold, determine the gesture corresponding to the current virtual hand;

[0094] (5) If the gesture matches the preset interactive gesture, the gesture control process is deemed to be successful, and the user is issuing control commands to the wearable device based on the gesture event.

[0095] In step S104, data interaction is performed between the wearable device and the terminal device according to the data interaction process and network protocol address.

[0096] For example, in this embodiment, after determining the data interaction process between the wearable device and the terminal device through the above steps, data interaction between the wearable device and the terminal device is achieved through a data relay network based on the network protocol address of the terminal device. For example, when the data interaction process involves the wearable device extracting the currently displayed data information from the terminal device, the wearable device sends the network protocol address and data request instruction to the network end. The network end determines the corresponding terminal device based on the network protocol address and sends the data request instruction to the terminal device. The terminal device then sends the data currently displayed on its screen to the network end based on the data request instruction, and the network end sends this data to the wearable device for data display. When the data interaction process involves the wearable device transmitting data to the terminal device for data display, the wearable device sends the currently displayed data and network protocol address to the network end. The currently displayed data is the data selected by the virtual hand in the wearable device and needs to be shared. The network end sends this data to the terminal device based on the network protocol address, so that the terminal device receives the data and displays it on its current screen.

[0097] The above method identifies terminal devices within a defined spatial range using wearable devices, generating virtual wireframes and network protocol addresses for these devices. In response to the detection of a user's palm within the defined spatial range, the wearable device identifies the user's palm and generates a virtual hand corresponding to it. If the distance between the virtual hand and the virtual wireframe is less than a distance threshold, and the target gesture of the virtual hand matches a preset interactive gesture, the data interaction flow between the wearable device and the terminal device is determined. Based on the data interaction flow and the network protocol address, data interaction occurs between the wearable device and the terminal device. Thus, by using wearable devices for gesture interaction, data interaction between devices is achieved, resulting in a more realistic interactive experience and more convenient operation.

[0098] Figure 5 This is a flowchart illustrating a data interaction method based on a wearable device according to an exemplary embodiment, such as... Figure 5 As shown, this method is applied to wearable devices and includes the following steps.

[0099] In step S201, the wearable device identifies the terminal devices within a set spatial range and generates a virtual wireframe and network protocol address for the terminal devices.

[0100] For example, in this embodiment, the method of generating the virtual wireframe and network protocol address is the same as in step S101 above, and can be referred to step S101 above, without further description.

[0101] In step S202, in response to detecting the user's palm within a set spatial range, the user's palm is identified to generate a virtual hand corresponding to the user's palm in the wearable device.

[0102] For example, the method of generating the virtual hand in this embodiment is the same as in step S102 above, and can be referred to step S102 above, without further description.

[0103] Optionally, in one embodiment, step S202 above includes:

[0104] Key points are identified on the user's palm to generate multiple key points corresponding to the user's palm.

[0105] Based on the user's palm displayed in the wearable device, multiple key points of the palm are connected to generate a virtual hand.

[0106] For example, in this embodiment, key points of the user's palm are identified based on the activity points of the user's palm. For instance, taking the index finger as an example, the activity points of the index finger are identified based on its movement characteristics, and palm key points corresponding to the index finger are generated at the positions where the index finger moves. Multiple palm key points are generated based on the same identification logic. It should be noted that in this embodiment, the generated position of the palm key points in the wearable device is close to the user's palm. From the user's perspective, the palm key points can be observed attached to the user's palm. Therefore, multiple palm key points can be connected according to the display position and outline of the user's palm in the wearable device to generate a virtual hand corresponding to the user's palm. Figure 6 This is a schematic diagram illustrating a virtual hand according to an exemplary embodiment, such as... Figure 6As shown, by identifying virtual points on the user's palm and connecting them according to the outline of the user's palm in the wearable device, a virtual hand is generated as shown in the figure. It is worth mentioning that in this embodiment, the virtual points are numbered according to their arrangement order, which facilitates subsequent recognition of the target gesture corresponding to the virtual hand, thereby analyzing the user's control intent. For example, Figure 7 This is a flowchart illustrating a target gesture recognition method according to an exemplary embodiment, such as... Figure 7 As shown, this method is applied to mixed reality glasses and includes the following steps.

[0107] (1) In response to the start of a scene in the mixed reality glasses, turn on the camera of the mixed reality glasses;

[0108] (2) Analyze the images captured by the camera through the mixed reality glasses to determine whether the user's palm is in the image;

[0109] (3) When the user's palm is present, generate key points corresponding to the user's palm in the space of the mixed reality glasses;

[0110] (4) Determine the gesture corresponding to the user's palm based on the location of the key point.

[0111] Alternatively, in another implementation, the interaction method further includes:

[0112] The movement trajectory of multiple key points in the virtual hand is monitored to generate the target gesture of the virtual hand.

[0113] If the similarity between the target gesture and the preset interactive gesture reaches a similarity threshold, the target gesture is determined to be a match for the preset interactive gesture.

[0114] It is worth mentioning that the virtual hand in the wearable device is positioned to fit the user's palm, and the virtual hand will move with the user's palm. Therefore, in this embodiment, the movement trajectory of multiple key points of the palm corresponding to the virtual hand is monitored. Based on the relative relationship between each key point of the palm, the target gesture corresponding to the user's palm is determined, and the target gesture is compared with the preset interactive gesture to determine the similarity between the two. When the similarity reaches the similarity threshold, it is determined that the target gesture matches the preset interactive gesture.

[0115] In step S203, the dynamic trajectory of the virtual hand is monitored to generate virtual hand gestures.

[0116] In step S204, if the gesture action is a set extraction action, the current extraction target of the virtual hand in the wearable device is determined.

[0117] It is worth mentioning that when wearable devices interact with terminal devices, users need to issue data interaction commands to the wearable device via their palm. Different data interaction processes all include data extraction actions, and the data interaction process can be determined based on the location of the data extraction action. For example, in this embodiment, the preset interaction gestures include setting an extraction action and setting a release action. The wearable device monitors the dynamic trajectory of each finger in the virtual hand to generate a virtual hand gesture. This gesture is compared with the preset interaction gestures. When the gesture is a set extraction action, the user needs to select the current extraction target on the wearable device's display.

[0118] In step S205, if the current target being extracted is not within the display range of the virtual wireframe, then the data interaction process is determined to be the terminal device display process.

[0119] In step S206, if the current target being extracted is within the display range of the virtual wireframe, then the data interaction process is determined to be a wearable device display process.

[0120] It is worth noting that the virtual wireframe generated in the above steps of this embodiment can be the boundary line of the terminal device. Each terminal device is equipped with a display screen, and typically the boundary line of the terminal device surrounds the display screen. Therefore, the display range of the virtual wireframe is the display screen area of ​​the terminal device. Thus, in this embodiment, the interaction flow between the wearable device and the terminal device is determined by whether the extraction target corresponding to the set extraction action is within the display range of the virtual wireframe. For example, if the current extraction target is within the display range of the virtual wireframe corresponding to the terminal device, it is determined that the user's intention is to extract data from the terminal device for display on the wearable device, and the corresponding data interaction flow is the wearable device display flow. If the current extraction target is not within the display range of the virtual wireframe corresponding to the terminal device, it is determined that the user's intention is to extract data from the wearable device to the terminal device for data display, and the corresponding data interaction flow is the terminal device display flow.

[0121] In step S207, data interaction is performed between the wearable device and the terminal device according to the data interaction process and network protocol address.

[0122] For example, the data interaction method in this embodiment is the same as in step S104 above, and can be referred to step S104 above, without further description.

[0123] Optionally, in one embodiment, step S207 includes:

[0124] In the case of a wearable device display process, a network protocol address is sent to the network end. The network protocol address is used to instruct the network end to request the first target data corresponding to the current extraction target from the terminal device according to the network protocol address.

[0125] In response to receiving the first target data from the network, the first target data is displayed at a designated location on the wearable device.

[0126] In the case where the data interaction process is a process of displaying the process to the terminal device, the second target data corresponding to the currently extracted target and the network protocol address are sent to the network end. The network protocol address is used to instruct the network end to send the second target data to the terminal device according to the network protocol address so that the second target data can be displayed on the terminal device.

[0127] Example, Figure 8 This is a schematic diagram illustrating a wearable device demonstration process according to an exemplary embodiment, such as... Figure 8 As shown, in this embodiment, when the data interaction process is determined to be a wearable device display process through the above steps, the wearable device sends a network protocol address to the network end. The network end sends a data request to the corresponding terminal device according to the network protocol address. After receiving the data request, the terminal device sends the first target data currently displayed on the screen to the network end. The network end relays the first target data back to the wearable device and displays the first target data at a set position on the wearable device display device, thereby realizing the synchronous display of the first target data in the wearable device and the terminal device. Figure 9 This is a schematic diagram illustrating a terminal device display process according to an exemplary embodiment, such as... Figure 9 As shown, when the data interaction process is determined to be a terminal device display process, the wearable device sends the second target data and network protocol address corresponding to the currently extracted target to the network end. After the network end determines the corresponding terminal device based on the network protocol address, it sends the second target data to the terminal device. After receiving the second target data, the terminal device displays the second target data on the current display screen.

[0128] In this way, after the user starts the wearable device, they can set interactive gestures on the wearable device to realize data transmission between the wearable device and the terminal device, and complete the synchronous display between the wearable device and the terminal device, making the interactive experience more realistic and the interactive operation more convenient.

[0129] Figure 10 This is a flowchart illustrating a data interaction method based on mixed reality glasses according to an exemplary embodiment, such as... Figure 10 As shown, this method is applied to mixed reality glasses and includes the following steps.

[0130] (1) Interact with the target gesture corresponding to the user's palm using mixed reality glasses;

[0131] (2) When a gesture interaction event is determined to exist, obtain the device IP address corresponding to the terminal device in the mixed reality glasses;

[0132] (3) Based on the interaction trajectory of the target gesture in the mixed reality glasses, determine the interaction process between the terminal device and the mixed reality glasses. For example, when the user's palm drags a file in the space of the mixed reality glasses to the virtual wireframe represented by the terminal device, the interaction process is determined to be that the mixed reality glasses transmit data to the terminal device for display; when the user's palm drags a file displayed in the APP of the terminal device to any space in the mixed reality glasses, the interaction process is determined to be that the mixed reality glasses request data from the terminal device and display the data in the corresponding position of the mixed reality glasses.

[0133] (4) The mixed reality glasses transmit information to the terminal device through the server according to the interaction process;

[0134] (5) When the interaction process is for the mixed reality glasses to transmit data to the terminal device, the mixed reality glasses send information to the server, which includes the target data currently selected by the user's palm and the device IP address corresponding to the terminal device; the server sends the target data to the terminal device according to the device IP address; after receiving the target data, the background APP of the terminal device displays the target data on the screen.

[0135] (6) When the interaction process is that the mixed reality glasses request data from the terminal device, the mixed reality glasses send the data request and the device IP address to the server. The server sends the data request to the corresponding terminal device according to the device IP address. The terminal device sends the corresponding target data to the server according to the data request. The server transmits the target data to the mixed reality glasses. After receiving the target data, the mixed reality glasses display the target data at the corresponding display position.

[0136] In this way, after the user starts the wearable device, they can set interactive gestures on the wearable device to realize data transmission between the wearable device and the terminal device, and complete the synchronous display between the wearable device and the terminal device, making the interactive experience more realistic and the interactive operation more convenient.

[0137] Figure 11 This is a block diagram illustrating a data interaction device based on a wearable device according to an exemplary embodiment. (Refer to...) Figure 11 The device 100 includes: a first generation module 110, a second generation module 120, a determination module 130, and an execution module 140.

[0138] The first generation module 110 is configured to identify terminal devices within a set spatial range based on wearable devices, and generate virtual wireframes and network protocol addresses for the terminal devices.

[0139] The second generation module 120 is configured to recognize the user's palm in response to detecting the user's palm within a set spatial range, so as to generate a virtual hand corresponding to the user's palm in the wearable device.

[0140] The determination module 130 is configured to determine the data interaction process between the wearable device and the terminal device when the distance between the virtual hand and the virtual wireframe is less than a distance threshold and the target gesture of the virtual hand matches the preset interactive gesture.

[0141] The execution module 140 is configured to perform data interaction between the wearable device and the terminal device according to the data interaction process and network protocol address.

[0142] Optionally, module 130 is configured as follows:

[0143] Monitor the dynamic trajectory of the virtual hand to generate virtual hand gestures;

[0144] When the gesture action is the set extraction action, determine the current extraction target of the virtual hand in the wearable device;

[0145] If the target being extracted is not within the display area of ​​the virtual wireframe, then the data interaction process is determined to be the terminal device display process;

[0146] If the target being extracted is within the display area of ​​the virtual wireframe, then the data interaction process is determined to be a wearable device display process.

[0147] Optionally, execution module 140 is configured as follows:

[0148] In the case of a wearable device display process, a network protocol address is sent to the network end. The network protocol address is used to instruct the network end to request the first target data corresponding to the current extraction target from the terminal device according to the network protocol address.

[0149] In response to receiving the first target data from the network, the first target data is displayed at a set location on the wearable device;

[0150] In the case where the data interaction process is a process of displaying the process to the terminal device, the second target data corresponding to the currently extracted target and the network protocol address are sent to the network end. The network protocol address is used to instruct the network end to send the second target data to the terminal device according to the network protocol address so that the second target data can be displayed on the terminal device.

[0151] Optionally, the second generation module 120 is configured as follows:

[0152] Key points are identified on the user's palm to generate multiple key points corresponding to the user's palm;

[0153] Based on the user's palm displayed in the wearable device, multiple key points of the palm are connected to generate a virtual hand.

[0154] Optionally, the device 100 includes a determination module, which is configured to:

[0155] The movement trajectory of multiple key points in the virtual hand is monitored to generate the target gesture of the virtual hand;

[0156] If the similarity between the target gesture and the preset interactive gesture reaches a similarity threshold, the target gesture is determined to be a match for the preset interactive gesture.

[0157] Optionally, the first generation module 110 is configured as follows:

[0158] Based on wearable devices, terminal devices within a defined spatial range are identified to generate the appearance features of the terminal devices;

[0159] Based on the appearance features, generate a virtual wireframe;

[0160] The virtual wireframe is sent to the network end, and the virtual wireframe is used to instruct the network end to determine the network protocol address of the terminal device based on the virtual wireframe.

[0161] The network protocol address received from the network end.

[0162] Optionally, the first generation module 110 is configured as follows:

[0163] The wearable device identifies terminal devices within a defined spatial range to determine the actual spatial distance between the wearable device and the terminal device.

[0164] The location information and actual spatial distance of the wearable device are reported to the network end. The location information and actual spatial distance are used to instruct the network end to determine the network protocol address of the terminal device based on the location information and actual spatial distance.

[0165] The network protocol address received from the network end.

[0166] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0167] This disclosure also provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the steps of the data interaction method based on a wearable device provided in this disclosure.

[0168] Figure 12 This is a block diagram illustrating a wearable device according to an exemplary embodiment. For example, device 1200 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0169] Reference Figure 12 The device 1200 may include one or more of the following components: processing component 1202, memory 1204, power supply component 1206, multimedia component 1208, audio component 1210, input / output interface 1212, sensor component 1214, and communication component 1216.

[0170] Processing component 1202 typically controls the overall operation of device 1200, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 1202 may include one or more processors 1220 to execute instructions to perform all or part of the steps of the aforementioned methods for data interaction based on wearable devices. Furthermore, processing component 1202 may include one or more modules to facilitate interaction between processing component 1202 and other components. For example, processing component 1202 may include a multimedia module to facilitate interaction between multimedia component 1208 and processing component 1202.

[0171] Memory 1204 is configured to store various types of data to support the operation of device 1200. Examples of this data include instructions for any application or method operating on device 1200, contact data, phonebook data, messages, pictures, videos, etc. Memory 1204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0172] Power supply component 1206 provides power to various components of device 1200. Power supply component 1206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 1200.

[0173] Multimedia component 1208 includes a screen that provides an output interface between the device 1200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 1208 includes a front-facing camera and / or a rear-facing camera. When the device 1200 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0174] Audio component 1210 is configured to output and / or input audio signals. For example, audio component 1210 includes a microphone (MIC) configured to receive external audio signals when device 1200 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 1204 or transmitted via communication component 1216. In some embodiments, audio component 1210 also includes a speaker for outputting audio signals.

[0175] Input / output interface 1212 provides an interface between processing component 1202 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, start buttons, and lock buttons.

[0176] Sensor assembly 1214 includes one or more sensors for providing status assessments of various aspects of device 1200. For example, sensor assembly 1214 may detect the on / off state of device 1200, the relative positioning of components such as the display and keypad of device 1200, changes in the position of device 1200 or a component of device 1200, the presence or absence of user contact with device 1200, the orientation or acceleration / deceleration of device 1200, and temperature changes of device 1200. Sensor assembly 1214 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 1214 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 1214 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0177] Communication component 1216 is configured to facilitate wired or wireless communication between device 1200 and other devices. Device 1200 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 1216 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 1216 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0178] In an exemplary embodiment, device 1200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described data interaction method based on wearable devices.

[0179] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1204 including instructions, which can be executed by a processor 1220 of device 1200 to complete the aforementioned data interaction method based on a wearable device. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0180] The aforementioned device can be a standalone electronic device or a part of a standalone electronic device. For example, in one embodiment, the device can be an integrated circuit (IC) or a chip, wherein the integrated circuit can be a single IC or a collection of multiple ICs. The chip can include, but is not limited to, the following types: GPU (Graphics Processing Unit), CPU (Central Processing Unit), FPGA (Field Programmable Gate Array), DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), and SoC (System on Chip). The aforementioned integrated circuit or chip can be used to execute executable instructions (or code) to implement the aforementioned data interaction method based on wearable devices. The executable instructions can be stored in the integrated circuit or chip or obtained from other devices or equipment. For example, the integrated circuit or chip includes a processor, memory, and an interface for communicating with other devices. The executable instructions can be stored in the memory, and when the executable instructions are executed by the processor, the above-mentioned data interaction method based on wearable devices can be implemented; or, the integrated circuit or chip can receive the executable instructions through the interface and transmit them to the processor for execution to implement the above-mentioned data interaction method based on wearable devices.

[0181] In another exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program executable by a programmable device, the computer program having a code portion for performing the above-described data interaction method based on a wearable device when executed by the programmable device.

[0182] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of this disclosure. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0183] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A data interaction method based on wearable devices, characterized in that, include: Based on the wearable device, the terminal devices within a set spatial range are identified, and a virtual wireframe and network protocol address of the terminal devices are generated; In response to detecting a user's palm within the defined spatial range, the user's palm is identified to generate a virtual hand corresponding to the user's palm in the wearable device; When the distance between the virtual hand and the virtual wireframe is less than a distance threshold, and the target gesture of the virtual hand matches a preset interactive gesture, a data interaction process between the wearable device and the terminal device is determined. The data interaction process is either a terminal device display process or a wearable device display process. The terminal device display process is either a process in which the wearable device transmits data to the terminal device for display, or a process in which the terminal device transmits data to the wearable device for display. Data interaction is performed between the wearable device and the terminal device according to the data interaction process and the network protocol address.

2. The interaction method according to claim 1, characterized in that, The process of determining the data interaction between the wearable device and the terminal device includes: Monitor the dynamic trajectory of the virtual hand to generate the gestures of the virtual hand; When the gesture action is a set extraction action, the current extraction target of the virtual hand in the wearable device is determined; If the current extraction target is not within the display range of the virtual wireframe, then the data interaction process is determined to be a terminal device display process; If the current extraction target is within the display range of the virtual wireframe, then the data interaction process is determined to be a wearable device display process.

3. The interaction method according to claim 2, characterized in that, The step of performing data interaction between the wearable device and the terminal device according to the data interaction process and the network protocol address includes: When the data interaction process is the wearable device display process, the network protocol address is sent to the network end. The network protocol address is used to instruct the network end to request the first target data corresponding to the current extraction target from the terminal device according to the network protocol address. In response to receiving the first target data fed back from the network, the first target data is displayed at a set location on the wearable device; When the data interaction process is the terminal device display process, the second target data corresponding to the current extraction target and the network protocol address are sent to the network end. The network protocol address is used to instruct the network end to send the second target data to the terminal device according to the network protocol address, so as to display the second target data in the terminal device.

4. The interaction method according to claim 1, characterized in that, The step of recognizing the user's palm to generate a virtual hand corresponding to the user's palm in the wearable device includes: Key point recognition is performed on the user's palm to generate multiple palm key points corresponding to the user's palm; Based on the user's palm displayed in the wearable device, the multiple key points of the palm are connected to generate the virtual hand.

5. The interaction method according to claim 4, characterized in that, The method includes: The movement trajectory of the multiple key points of the palm in the virtual hand is monitored to generate the target gesture of the virtual hand; If the similarity between the target gesture and the preset interactive gesture reaches a similarity threshold, the target gesture is determined to match the preset interactive gesture.

6. The interaction method according to claim 1, characterized in that, The step of identifying terminal devices within a defined spatial range based on the wearable device and generating a virtual wireframe and network protocol address for the terminal devices includes: The wearable device is used to identify the terminal device within a set spatial range to generate the appearance features of the terminal device. The virtual wireframe is generated based on the described appearance features; The virtual wireframe is sent to the network end, and the virtual wireframe is used to instruct the network end to determine the network protocol address of the terminal device based on the virtual wireframe. Receive the network protocol address fed back by the network terminal.

7. The interaction method according to claim 1, characterized in that, The step of identifying terminal devices within a defined spatial range based on the wearable device and generating the network protocol address of the terminal devices includes: The wearable device identifies terminal devices within a defined spatial range to determine the actual spatial distance between the wearable device and the terminal devices. The location information and actual spatial distance of the wearable device are reported to the network terminal. The location information and actual spatial distance are used to instruct the network terminal to determine the network protocol address of the terminal device based on the location information and actual spatial distance. Receive the network protocol address fed back by the network terminal.

8. A data interaction device based on a wearable device, characterized in that, include: The first generation module is configured to identify terminal devices within a set spatial range based on the wearable device, and generate virtual wireframes and network protocol addresses for the terminal devices. The second generation module is configured to, in response to detecting a user's palm within the set spatial range, identify the user's palm to generate a virtual hand corresponding to the user's palm in the wearable device; The determination module is configured to determine the data interaction process between the wearable device and the terminal device when the distance between the virtual hand and the virtual wireframe is less than a distance threshold and the target gesture of the virtual hand matches a preset interactive gesture. The data interaction process is either a terminal device display process or a wearable device display process. The terminal device display process is either a process in which the wearable device transmits data to the terminal device for display or a process in which the terminal device transmits data to the wearable device for display. The execution module is configured to perform data interaction between the wearable device and the terminal device according to the data interaction process and the network protocol address.

9. A wearable device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the steps of the method according to any one of claims 1-7 when executing the executable instructions.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 7.

Images