Gesture recognition method, device and system

Abstract

The invention discloses a gesture recognition method, device and system. The method comprises the steps that a first device obtains a gesture list supported by a second device; the first device receives virtual reality VR data sent by the second device, and displays the VR data; the first device collects and recognizes a gesture, a gesture message is generated based on the obtained gesture list supported by the second device, and the gesture message carries at least one of the following information: gesture information and control information; and the first device sends the gesture message to the second device.

Description

[0001] This application is a divisional application of the application with the application date of "June 30, 2017", application number of "201710523769.5", and application title of "A Gesture Recognition Method, Device, and System". Technical Field

[0002] The present invention relates to the field of communication and information, and particularly to a gesture recognition method, device, and system for virtual reality (VR). Background Art

[0003] The development of digital devices and networks has added a lot of convenience to people's daily lives. For example, in the field of digital multimedia, current technologies enable people to watch high-definition TV at home. The sources of TV programs may come from digital optical discs, cable TV, the Internet, etc., and people can experience stereo, 5.1-channel, 7.1-channel, and even more realistic sound effects. Moreover, people can also use tablet electronic devices (PADs) and mobile phones to achieve these experiences. In addition, people can transfer the playback of digital content between different devices through the network and control the playback of a multimedia device through a remote control or gestures, such as controlling the switching of the previous and next channel programs. Some air conditioners and refrigerators also provide network control functions.

[0004] Gesture control is a relatively novel device control method at present: a camera on a device monitors gesture actions, analyzes and recognizes them, and finally converts them into control of the device; or the user uses a wearable device, such as wearing devices similar to rings, watches, vests, etc. on the hand, arm, and body to recognize the user's actions, and can also achieve control of the device. Using gesture control technology, some products enable people to control devices through gestures. For example, a camera is added to a TV to collect and recognize human gestures, and then according to the pre-defined correspondence between gestures and control commands, the effect of gesture control of the TV is achieved. The implemented controls include changing channels, adjusting the volume, etc.

[0005] In gesture recognition technology, one way is to require the device to be controlled to have a camera for computer vision recognition. In some environments (such as the home environment), there may be devices that do not have a camera, for example, due to device size or cost reasons. However, these devices also hope to be operable by the user through gestures, and hope that another camera can convey gesture information to this device, so as to achieve the user's control of this device through gestures.

[0006] To this end, based on the technology of transmitting control information between different devices through a network, the controlled device publishes a list of supported gestures and / or a list of instructions corresponding to the supported gestures in the network; the gesture acquisition device obtains the list of supported gestures and / or the list of instructions corresponding to the supported gestures of the controlled device through the network. The gesture acquisition device acquires a gesture and sends the acquired gesture to the controlled device. When implementing the above solution, the user needs to face the controlled device. Otherwise, the user cannot see the result of the control operation on the device, which is not convenient to use. Summary of the Invention

[0007] To solve the above technical problems, embodiments of the present invention provide a gesture recognition method, device, and system.

[0008] The gesture recognition method provided by the embodiments of the present invention includes:

[0009] The first device obtains a list of gestures supported by the second device;

[0010] The first device receives VR data sent by the second device and displays the VR data;

[0011] The first device acquires and recognizes a gesture, generates a gesture message based on the obtained list of gestures supported by the second device, and the gesture message carries at least one of the following information: gesture information, control information;

[0012] The first device sends the gesture message to the second device.

[0013] In the embodiments of the present invention, the first device obtains a list of gestures supported by the second device, including:

[0014] The first device receives a subscription gesture message sent by the second device, and the subscription gesture message carries the list of gestures supported by the second device.

[0015] In the embodiments of the present invention, before the first device receives the subscription gesture message sent by the second device, the method further includes:

[0016] The first device sends a query broadcast or multicast message to discover the second device; or, after the first device receives a query broadcast or multicast message sent by the second device, it responds to the query broadcast or multicast message;

[0017] After the first device and the second device complete device discovery, the first device and the second device establish a network connection.

[0018] In the embodiments of the present invention, before the first device receives the subscription gesture message sent by the second device, the method further includes:

[0019] The first device establishes a network connection with the second device through the network address of the second device.

[0020] In an embodiment of the present invention, the gesture list includes at least one of the following: the gesture number supported by the second device, the gesture feature data corresponding to the gesture number, and the gesture control instruction corresponding to the gesture number.

[0021] In an embodiment of the present invention, the gesture information includes at least one of the following: the gesture number, the gesture parameter, the current image parameter when the gesture is collected, the position mapping parameter of the gesture on the current image, the continuously collected gesture video stream, and the predefined instruction code.

[0022] In an embodiment of the present invention, the second device stores at least one of the following information: the VR data of the second electronic device, the gesture list supported by the second device, where the VR data is VR image data or VR video data;

[0023] The subscription gesture message also carries the network address or network port corresponding to the VR data; the first device receives the VR data sent by the second device according to the network address or network port corresponding to the VR data.

[0024] A gesture recognition method provided by another embodiment of the present invention includes:

[0025] The second device sends the gesture list supported by the second device to the first device;

[0026] The second device sends VR data to the first device so that the first device can display the VR data;

[0027] The second device receives the gesture message sent by the first device, and the gesture message carries at least one of the following information: gesture information, control information; and executes the corresponding control instruction according to the gesture message.

[0028] In an embodiment of the present invention, the second device sending the gesture list supported by the second device to the first device includes:

[0029] The second device sends a subscription gesture message to the first device, and the subscription gesture message carries the gesture list supported by the second device.

[0030] In an embodiment of the present invention, before the second device sends a subscription gesture message to the first device, the method further includes:

[0031] The second device sends a query broadcast or multicast message to perform device discovery on the first device; or, after the second device receives the query broadcast or multicast message sent by the first device, it responds to the query broadcast or multicast message;

[0032] After the second device completes device discovery with the first device, the second device establishes a network connection with the first device.

[0033] In an embodiment of the present invention, before the second device sends a subscription gesture message to the first device, the method further includes:

[0034] The second device establishes a network connection with the first device through the network address of the first device.

[0035] In an embodiment of the present invention, the gesture list includes at least one of the following: the gesture numbers supported by the second device, the gesture feature data corresponding to the gesture numbers, and the gesture control instructions corresponding to the gesture numbers.

[0036] In an embodiment of the present invention, the gesture information includes at least one of the following: gesture number, gesture parameter, current image parameter when the gesture is collected, position mapping parameter of the gesture on the current image, continuously collected gesture video stream, and predefined instruction code.

[0037] In an embodiment of the present invention, the second device stores at least one of the following information: VR data of the second electronic device, gesture list supported by the second device, where the VR data is VR image data or VR video data;

[0038] The subscription gesture message also carries the network address or network port corresponding to the VR data.

[0039] The gesture recognition device provided in an embodiment of the present invention is applied to the first device, and the device includes:

[0040] A receiving unit, configured to obtain the gesture list supported by the second device; receive the VR data sent by the second device;

[0041] A display unit, configured to display the VR data;

[0042] A collection and recognition unit, configured to collect and recognize gestures, and generate a gesture message based on the obtained gesture list supported by the second device, where the gesture message carries at least one of the following information: gesture information, control information;

[0043] A sending unit, configured to send the gesture message to the second device.

[0044] In an embodiment of the present invention, the receiving unit is further configured to receive a subscription gesture message sent by the second device, and the subscription gesture message carries the gesture list supported by the second device.

[0045] In an embodiment of the present invention, the device further includes:

[0046] A connection unit, configured to send out a query broadcast or multicast message for device discovery of a second device; or, after the first device receives a query broadcast or multicast message sent by the second device, respond to the query broadcast or multicast message; after the first device and the second device complete device discovery, the first device and the second device establish a network connection.

[0047] In an embodiment of the present invention, the apparatus further includes:

[0048] A connection unit, configured to establish a network connection with the second device through the network address of the second device.

[0049] In an embodiment of the present invention, the gesture list includes at least one of the following: the gesture numbers supported by the second device, the gesture feature data corresponding to the gesture numbers, and the gesture control instructions corresponding to the gesture numbers.

[0050] In an embodiment of the present invention, the gesture information includes at least one of the following: gesture numbers, gesture parameters, current image parameters when the gesture is collected, position mapping parameters of the gesture on the current image, continuously collected gesture video streams, and predefined instruction codes.

[0051] In an embodiment of the present invention, at least one of the following information is stored in the second device: VR data of the second electronic device, a gesture list supported by the second device, where the VR data is VR image data or VR video data;

[0052] The subscription gesture message further carries the network address or network port corresponding to the VR data; the first device receives the VR data sent by the second device according to the network address or network port corresponding to the VR data.

[0053] A gesture recognition apparatus provided in another embodiment of the present invention is applied to a second device, and the apparatus includes:

[0054] A sending unit, configured to send a gesture list supported by the second device to the first device; send VR data to the first device to enable the first device to display the VR data;

[0055] A receiving unit, configured to receive a gesture message sent by the first device, where the gesture message carries at least one of the following information: gesture information, control information;

[0056] An execution unit, configured to execute corresponding control instructions according to the gesture message.

[0057] In an embodiment of the present invention, the sending unit is further configured to send a subscription gesture message to the first device, and the subscription gesture message carries the gesture list supported by the second device.

[0058] In an embodiment of the present invention, the device further includes:

[0059] A connection unit, configured to send a query broadcast or multicast message to perform device discovery on a first device; or, after the second device receives a query broadcast or multicast message sent by the first device, respond to the query broadcast or multicast message; after the second device and the first device complete device discovery, the second device establishes a network connection with the first device.

[0060] In an embodiment of the present invention, the device further includes:

[0061] A connection unit, configured to establish a network connection with the first device through the network address of the first device.

[0062] In an embodiment of the present invention, the gesture list includes at least one of the following: the gesture numbers supported by the second device, the gesture feature data corresponding to the gesture numbers, and the gesture control instructions corresponding to the gesture numbers.

[0063] In an embodiment of the present invention, the gesture information includes at least one of the following: gesture numbers, gesture parameters, current image parameters when the gesture is collected, position mapping parameters of the gesture on the current image, continuously collected gesture video streams, and predefined instruction codes.

[0064] In an embodiment of the present invention, the second device stores at least one of the following information: VR data of the second electronic device, a gesture list supported by the second device, where the VR data is VR image data or VR video data;

[0065] The subscribed gesture message also carries the network address or network port corresponding to the VR data.

[0066] The gesture recognition system provided by an embodiment of the present invention includes: a first device and a second device; where

[0067] The first device is configured to obtain a gesture list supported by the second device; receive VR data sent by the second device and display the VR data; collect and recognize gestures, generate a gesture message based on the obtained gesture list supported by the second device, where the gesture message carries at least one of the following information: gesture information, control information; and send the gesture message to the second device;

[0068] The second device is configured to send the gesture list supported by the second device to the first device; send VR data to the first device so that the first device displays the VR data; receive the gesture message sent by the first device, where the gesture message carries at least one of the following information: gesture information, control information; and execute corresponding control instructions according to the gesture message.

[0069] In an embodiment of the present invention, the gesture list includes at least one of the following: the gesture numbers supported by the second device, the gesture feature data corresponding to the gesture numbers, and the gesture control instructions corresponding to the gesture numbers.

[0070] In an embodiment of the present invention, the gesture information includes at least one of the following: gesture numbers, gesture parameters, current image parameters when the gesture is collected, position mapping parameters of the gesture on the current image, continuously collected gesture video streams, and predefined instruction codes.

[0071] In the technical solution of the embodiment of the present invention, the first device obtains the gesture list supported by the second device; the first device receives the VR data sent by the second device and displays the VR data; the first device collects and recognizes gestures, generates a gesture message based on the obtained gesture list supported by the second device, and the gesture message carries at least one of the following information: gesture information, control information; the first device sends the gesture message to the second device. By adopting the technical solution of the present invention, the user does not have to face the target device to be operated, so that a device without a gesture recognition component can also be gesture-controlled, the operation is more automated and more user-friendly, people can conveniently use gestures to control the device to be controlled, which not only makes people's lives easier and more enjoyable, but also makes it easy to understand and get started when mastering this function and using such devices, and also reduces the costs of all parties in the industrial chain. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] The drawings generally illustrate, by way of example and not limitation, the various embodiments discussed herein.

[0073] Figure 1 It is a schematic diagram of the scenario of the embodiment of the present invention;

[0074] Figure 2 It is a schematic diagram of the VR headset device of the embodiment of the present invention;

[0075] Figure 3 It is a schematic diagram of the VR headset device and the controlled device of the embodiment of the present invention;

[0076] Figure 4 It is a schematic flow chart of the gesture recognition method of the embodiment of the present invention Figure 1 ;

[0077] Figure 5 It is a schematic flow chart of the gesture recognition method of the embodiment of the present invention Figure 2 ;

[0078] Figure 6 It is a schematic diagram of the message interaction of the embodiment of the present invention;

[0079] Figure 7Schematic diagram of event-triggered actions of the VR headset device according to an embodiment of the present invention;

[0080] Figure 8 Schematic flow chart of the VR headset device according to an embodiment of the present invention;

[0081] Figure 9 Schematic flow chart of the controlled device according to an embodiment of the present invention;

[0082] Figure 10 Schematic diagram of the structural composition of the gesture recognition device according to an embodiment of the present invention Figure 1 ;

[0083] Figure 11 Schematic diagram of the structural composition of the gesture recognition device according to an embodiment of the present invention Figure 2 ;

[0084] Figure 12 Schematic diagram of the structural composition of the gesture recognition system according to an embodiment of the present invention. Detailed implementation manners

[0085] In order to more comprehensively understand the features and technical content of the embodiments of the present invention, the implementation of the embodiments of the present invention will be described in detail below with reference to the accompanying drawings. The accompanying drawings are only for reference and illustration purposes and are not intended to limit the embodiments of the present invention.

[0086] Figure 1 Schematic diagram of the scenario according to an embodiment of the present invention. As Figure 1 shown, the embodiments of the present invention involve two devices, namely: the first device and the second device. Among them, the first device specifically refers to a VR display and gesture recognition device (subsequently referred to as a VR headset device); the second device specifically refers to a controlled device, such as a television.

[0087] In Figure 1 the scenario shown, the television supports the gesture control function but does not have a camera for collecting gestures; physically, the television may not be within the visual capture range of the VR headset device. For example, the user wears the VR headset device in the living room and is ready to control the television placed in the bedroom. In addition, in Figure 1 the scenario shown, only one controlled device is drawn. In actual applications, the VR headset device can control more than one controlled device, and it is not necessary for the controlled device to be within the visual capture range of the VR headset device, which provides more convenience for the user to control the controlled device.

[0088] In Figure 1In the described scenario, both of the two devices have network interfaces. For example, the VR headset device and the controlled device support IEEE 802.11b / g / n or IEEE 802.3, so that they can be connected to the IP network. Each device contains a control module that can control the device to send messages, receive messages, process manipulation instructions, or forward manipulation instructions to other devices.

[0089] The ability of the VR headset device and the controlled device to discover each other, connect, send messages, and receive messages on the network can be implemented using the Universal Plug and Play (UPnP) technology, or using the multicast Domain Name System (mDNS) and the Domain Name System-Service Discovery (DNS-SD) technology. These technologies are used in the IP network to respond to queries and provide function calls in the form of unicast and multicast queries according to a predefined message format. For example, the UPnP technology specifies how media display devices (such as TVs) and servers (such as DVD players and home storage servers) respond to queries and what call functions they provide. In addition, it is also possible to directly search for a certain device by entering the IP address, domain name, etc. of the destination device in the local device. For example, enter the IP address of the TV on the keyboard connected to the VR headset device to establish a network connection between the VR headset device and the TV.

[0090] In one embodiment, in order to support gesture control, the TV stores a list of gestures supported by the device, including: the supported gesture numbers, for example, 1 represents spreading out five fingers, 2 represents two fingers, 10 represents a fist, 20 represents waving an arm, etc.; it may also include: the control meanings corresponding to the gestures, for example, 1 represents turning on the power, 10 represents turning on the power, etc.; it may also include: gesture parameters, such as the duration of the gesture action, etc. When the TV receives these gestures, it performs corresponding functional operations.

[0091] Figure 2 Schematic diagram of the VR headset device according to an embodiment of the present invention. As Figure 2As shown in the figure, the VR headset device is similar to a common VR headset in terms of appearance, and includes: (1) an image display that can display VR images / videos in the form of a liquid crystal display or projection; (2) a camera with the ability to collect images / videos, as well as an infrared ranging module, and the collected images / videos and position data can be used for gesture recognition; in actual application, more than one camera can be assembled, for example, a camera for recognizing the eye movements of the user is also included; (3) a network module that provides network interconnection functions with other devices, such as a network connection with a controlled device; (4) a gyroscope that is used to measure the head movement data of the user, and these data are used to judge the head movements of the user, such as turning the head, lowering the head, raising the head, etc. In addition, the VR headset device also includes some structural components to facilitate the user to wear it.

[0092] In this embodiment, the VR headset device can recognize gestures within its visual range. The implementation method is that the camera and the ranging module collect images in its three-dimensional space, and analyze the collected images to recognize gestures.

[0093] As another embodiment, the VR headset device can also be connected to a wearable device to collect and recognize gestures, such as a ring-shaped device worn on the hand, a glove device worn on the hand, or a watch-type device worn on the arm. This wearable device can recognize the finger and arm movements of the user, and these movements are transmitted to the VR headset device as gestures.

[0094] It should be noted that, for the convenience of understanding, the embodiments of the present invention use the VR headset device as an example of the present invention. In fact, the head movement tracking module, the eye tracking module, the content display module, the gesture recognition module, the network module, various structural modules, etc. can completely support the functions of the embodiments of the present invention through other combinations. For example, an independent camera tracks and recognizes head movements and gesture movements without being worn on the user's head, and the content display module is installed in a spherical space separately.

[0095] Figure 3 This is a schematic diagram of the VR headset device and the controlled device according to the embodiment of the present invention. Figure 3 Respectively show the functional modules of the controlled device and the VR headset device. Among them:

[0096] Both the controlled device and the VR headset device have network units. The network units support, for example, IEEE 802.11b / g / n, or support IEEE 802.3, so that the devices can be connected to the IP network. In the embodiments of the present invention, the two devices are connected to each other through the network units. Each network unit contains a control module that can control the local device to send and receive messages with other devices.

[0097] The network unit provides the device with the capabilities of mutual discovery, connection, message sending and receiving on the network. It can be implemented using existing UPnP technology or mDNS and DNS-SD technology. This type of technology is used in IP networks to respond to queries and provide function calls in the form of unicast and multicast queries according to a predefined message format. For example, the UPnP technology specifies how media display devices (such as TVs) and servers (such as DVD players and home storage servers) respond to queries and what call functions they provide.

[0098] The controlled device includes a storage unit that stores a list of gestures supported by the controlled device and also stores VR images. The VR images can be panoramic 360° appearance images of the controlled device, can be an image file, or can be multiple files. The VR images will be transmitted from the controlled device to the VR headset device through the network unit function.

[0099] After receiving the gesture message, the control execution unit in the controlled device performs predefined function operations, such as changing channels, adjusting volume, adjusting air conditioner temperature, and so on.

[0100] The visual data acquisition unit in the VR headset device is used to capture and collect the gesture images (including video clips) of the user; the visual recognition unit stores algorithm programs and can recognize gestures from the gesture images, including recognizing gesture shapes, gesture position information, gesture movement directions, and so on. The image display unit in the VR headset device is used to display the VR images on the screen or project them in front of the user's eyes after receiving the VR images, providing a visual reference for the user's gesture operations. For example, it displays a front view of a TV set, facilitating the user to point to relevant areas of the TV set, make gestures, and express the intention of controlling the TV set. The position detection unit in the VR headset device detects the movement of the user's head and eyes, recognizes the user's viewing point, for the image display unit to adjust the display of the VR images, and is transmitted to the controlled device as part of the gesture information. The storage unit in the VR headset device can store the list of gestures subscribed by the received controlled device and cache VR images, etc.

[0101] Based on the specific functions of the above devices, the gesture recognition method of the embodiments of the present invention is proposed below.

[0102] Figure 4 Flow schematic of the gesture recognition method of the embodiments of the present invention Figure 1 As Figure 4 shown, the method includes the following steps:

[0103] Step 401: The first device obtains the list of gestures supported by the second device.

[0104] Specifically, the first device receives a subscription gesture message sent by the second device, and the subscription gesture message carries a list of gestures supported by the second device.

[0105] In an embodiment, before the first device receives the subscription gesture message sent by the second device, the method further includes:

[0106] The first device sends a query broadcast or multicast message to perform device discovery on the second device; or, after the first device receives a query broadcast or multicast message sent by the second device, it responds to the query broadcast or multicast message;

[0107] After the first device and the second device complete device discovery, the first device establishes a network connection with the second device.

[0108] In another embodiment, before the first device receives the subscription gesture message sent by the second device, the method further includes:

[0109] The first device establishes a network connection with the second device through the network address of the second device.

[0110] In an embodiment of the present invention, the gesture list includes at least one of the following: the gesture number supported by the second device, the gesture feature data corresponding to the gesture number, and the gesture control instruction corresponding to the gesture number.

[0111] Step 402: The first device receives the VR data sent by the second device and displays the VR data.

[0112] Step 403: The first device recognizes a gesture and generates a gesture message based on the obtained list of gestures supported by the second device. The gesture message carries at least one of the following information: gesture information, control information.

[0113] In an embodiment, the gesture information includes at least one of the following: gesture number, gesture parameter, current image parameter when the gesture is collected, position mapping parameter of the gesture on the current image, continuously collected gesture video stream, predefined instruction code.

[0114] In an embodiment of the present invention, according to the VR image parameters corresponding to the moment when the gesture image is collected and the position mapping parameter of the gesture image in the VR image, the second device can calculate the virtual image corresponding to the gesture based on this information and update it to the VR image and send it to the first device for display, enhancing the user's operation experience.

[0115] Step 404: The first device sends the gesture message to the second device.

[0116] In an embodiment of the present invention, the second device stores at least one of the following information: VR data of the second electronic device, a gesture list supported by the second device, where the VR data is VR image data or VR video data;

[0117] The subscription gesture message further carries the network address or network port corresponding to the VR data; the first device receives the VR data sent by the second device according to the network address or network port corresponding to the VR data. Wherein, the network address or network port corresponding to the VR data may be different from the network address or network port corresponding to the subscription gesture message.

[0118] Figure 5 Schematic flow of the gesture recognition method according to an embodiment of the present invention Figure 2 , such as Figure 5 shown, the method includes the following steps:

[0119] Step 501: The second device sends the gesture list supported by the second device to the first device.

[0120] Specifically, the second device sends a subscription gesture message to the first device, and the subscription gesture message carries the gesture list supported by the second device.

[0121] In one embodiment, before the second device sends the subscription gesture message to the first device, the method further includes:

[0122] The second device sends a query broadcast or multicast message to perform device discovery on the first device; or, after the second device receives the query broadcast or multicast message sent by the first device, it responds to the query broadcast or multicast message;

[0123] After the second device and the first device complete device discovery, the second device and the first device establish a network connection.

[0124] In another embodiment, before the second device sends the subscription gesture message to the first device, the method further includes:

[0125] The second device establishes a network connection with the first device through the network address of the first device.

[0126] In an embodiment of the present invention, the gesture list includes at least one of the following: the gesture number supported by the second device, the gesture feature data corresponding to the gesture number, the gesture control instruction corresponding to the gesture number.

[0127] Step 502: The second device sends the VR data to the first device so that the first device displays the VR data.

[0128] At least one of the following information is stored in the second device: VR data of the second electronic device, a list of gestures supported by the second device, where the VR data is VR image data or VR video data;

[0129] The subscription gesture message also carries the network address or network port corresponding to the VR data.

[0130] Step 503: The second device receives the gesture message sent by the first device, and the gesture message carries at least one of the following information: gesture information, control information; execute corresponding control instructions according to the gesture message.

[0131] In an embodiment, the gesture information includes at least one of the following: gesture number, gesture parameter, current image parameter when the gesture is collected, position mapping parameter of the gesture on the current image, continuously collected gesture video stream, predefined instruction code.

[0132] The gesture recognition method according to the embodiment of the present invention will be further described in detail below in combination with specific applications.

[0133] Figure 6 It is a schematic diagram of message interaction according to an embodiment of the present invention; in this embodiment, the controlled device sends a list of gestures supported by the controlled device to the VR headset device. As Figure 6 shown, the processing flow of this example includes:

[0134] 1) The controlled device and the VR headset device complete device discovery and connection.

[0135] Specifically, device discovery and connection can be achieved through methods such as UPnP, mDNS, manual input of IP search, etc. Finally, the controlled device and the VR headset device establish a network connection.

[0136] 2) The controlled device sends a subscription message to the VR headset device to subscribe to the gesture information of the VR headset device.

[0137] Here, the subscription message is a network message, and the message content is a gesture capability list. Examples of the gesture capability list include: supported gesture numbers, for example, 1 means spreading five fingers, 2 means two fingers, 10 means making a fist, 20 means waving an arm, etc.; it can also include: control meanings corresponding to gestures, for example, 1 means power on, 10 means power on, etc.; it can also include: gesture parameters, such as the duration of the gesture action, etc. When the controlled device receives these gestures, it performs corresponding function operations.

[0138] In an example, the subscription message also includes information such as the communication address and port for sending VR images.

[0139] In one example, the VR headset device responds to the controlled device and confirms the supported gestures. The VR headset device responds to the controlled device and sends a message including a list of gesture capabilities to the controlled device in response to the subscription request of the controlled device.

[0140] In one example, the gesture information subscribed by the controlled device to the VR headset device can also specify including a video stream, that is, it requests the VR headset device to collect the relevant video stream.

[0141] 3) The controlled device sends a VR image to the VR headset device.

[0142] After receiving the VR image, the VR headset device displays the VR content to the user. During the process of the user watching the VR image, the user can make gestures, and the VR headset device captures and recognizes the user's gestures. Thus, the user can make gestures as if facing a physical entity without having to face the physical entity of the controlled device.

[0143] After the VR headset device completes step 3), it is ready to receive information such as the communication address and port of the VR image. The controlled device, after receiving the subscription response message from the VR headset device, sends a VR image to the VR headset device. For the sake of efficiency and bandwidth, different network addresses or different ports are used for transmitting the subscription message and sending the VR image.

[0144] In this embodiment, the VR image technology includes: panoramic image technology, technology for tracking the movement of the user's head, and image display technology for the user's eye viewpoints.

[0145] 4) The VR headset device recognizes the user's gestures.

[0146] The VR headset device uses the equipped camera or wearable device to capture the user's gestures and recognize them as computer representations. After successful recognition, it is compared with the subscribed gesture list. If the recognized gesture is in the subscribed gesture list, it proceeds to the next step; otherwise, it continues to capture and recognize gestures.

[0147] 5) The VR headset device sends gesture information to the controlled device.

[0148] Among them, the gesture information includes the following content:

[0149] The supported gesture numbers, for example, 1 represents an open palm, 2 represents two fingers, 10 represents a fist, 20 represents waving an arm, etc.;

[0150] The control meanings corresponding to the gestures, for example, 1 represents power on, 10 represents power on, etc.;

[0151] Gesture parameters, such as the duration of the gesture action, etc.

[0152] Current image parameters: When capturing gestures, it refers to the image position data currently displayed to the user, such as the coordinate position of the top-left pixel of the current frame image in the entire panoramic image and the image size data.

[0153] Optionally, it may also include the position mapping parameters of the gesture on the current image.

[0154] The sent gesture information may also include a continuously captured video stream for other analyses such as gesture analysis, gesture position analysis, and body posture analysis by the controlled device.

[0155] In addition to the above gesture information being sent between the controlled device and the VR headset device, as another embodiment, the VR headset device can pre-define gestures and map them to instruction codes, and this mapping can also be re-specified by the user; for example, mapping a thumb to represent "OK", a palm with all fingers raised to represent "Cancel", an index finger to represent "1", and so on. The controlled device can directly subscribe to these mappings, thereby improving the control experience.

[0156] 6) If the user continuously makes gestures, the VR headset device continues to send gesture information.

[0157] 7) If the controlled device decides to end the gesture control process, it sends an end subscription gesture information to the VR headset device to end the gesture control, disconnect the connection between the two, and also includes ending the display of the VR image. Thus, the entire process ends normally.

[0158] Figure 7 It is a schematic diagram of event triggering actions of the VR headset device according to an embodiment of the present invention. As Figure 7 shown: After the VR headset device is started, through the event trigger program function, it executes the functions given in the embodiment of the present invention. Here, the events include: receiving a VR image (stream), detecting a gesture, and detecting head / eyeball (pupil) movement. After detecting an event, refer to Figure 6 , the VR headset device triggers the program to execute the following functions: displaying / playing the VR image; recognizing gestures, calculating gesture information, sending gesture information; adjusting the display / playing of the VR image, etc.

[0159] Figure 8 It is a flowchart of the VR headset device according to an embodiment of the present invention. This embodiment describes the flowchart of the VR headset device working. As Figure 8 shown, the process of this example includes:

[0160] 1) Device discovery and connection.

[0161] The VR headset device can first send out a query broadcast or multicast message; it can also respond to the broadcast or multicast message after receiving the query broadcast or multicast message from the controlled device in the network. After completing the discovery function, it can actively or passively establish a connection with the controlled device.

[0162] After establishing the connection, as an embodiment, the VR headset device can act as an HTTP client and send an HTTP request to the controlled device acting as an HTTP server in a RESTful manner. Of course, those skilled in the art should understand that it is also possible to set the controlled device as the HTTP client and the VR headset device as the HTTP server.

[0163] 2) Receive subscription gesture information.

[0164] As an embodiment, after establishing the HTTP connection, the VR headset device sends a GET request message to the controlled device to query the gesture list of the controlled device. The controlled device returns a response message, and the VR headset device obtains the subscription gesture information by receiving the response message.

[0165] The subscription gesture information here includes a set of gesture description information, referring to the foregoing embodiments.

[0166] In this step, the controlled device also sends the address and port for subsequent sending of virtual reality images to the VR headset device. The VR headset device receives and saves this information. The VR headset device establishes a network connection to the address and port for sending VR real images. Receiving VR images and receiving subscription gesture information can use different network ports.

[0167] 3) Receive VR images.

[0168] As an implementation method, the protocol for transmitting VR images can be a streaming media transmission protocol such as HLS, RSTP, DASH, or downloading files through HTTP.

[0169] 4) Display VR images.

[0170] The VR headset device presents the received VR images to the user through a display screen or a projection component, or presents the image content that needs to be updated to the user.

[0171] 5) If head / pupil movement is detected, go to step 4).

[0172] The VR headset device calculates the user's pose based on the detection data and determines the image to be displayed to the user. For example, as the user looks up / eyeballs move up, the content of the upper part of the image is displayed to the user.

[0173] 6) Collect and recognize gesture information.

[0174] The VR headset device captures the user's gestures through a camera, wearable device, or other computer vision acquisition components, analyzes and recognizes the gestures, and thus obtains gesture information.

[0175] 7) Send the gesture information.

[0176] Among them, the gesture information includes:

[0177] The supported gesture numbers, for example, 1 means fingers spread, 2 means two fingers, 10 means a fist, 20 means waving the arm, etc.;

[0178] The control meaning corresponding to the gesture, for example, 1 means power on, 10 means power on, etc.;

[0179] Gesture parameters, such as the duration of the gesture action, etc.

[0180] Current image parameters: When capturing the gesture, the current image position data displayed to the user, such as the coordinate position of the upper left pixel point of the current frame image in the entire panoramic image, and the image size data.

[0181] Optionally, it includes the position mapping parameters of the gesture on the current image.

[0182] As an embodiment, the VR headset device can send the gesture information to the controlled device using HTTP PUT or HTTP POST messages.

[0183] 8) If the instruction message from the controlled device is not received, go to step 5).

[0184] After the VR headset device receives the response message from the controlled device, if it is analyzed that there is no further instruction information, it continues to go to step 5 to continue detecting the user's behavior and recognizing gestures.

[0185] 9) If the instruction information from the controlled device is received, if the instruction information from the controlled device indicates to end the gesture recognition, end the process, otherwise go to step 3).

[0186] Figure 9 It is a schematic flow diagram of the controlled device in the embodiment of the present invention. This embodiment describes the working flow chart of the controlled device. As Figure 9 shown, the process of this example includes:

[0187] 1) Device discovery and connection.

[0188] The controlled device can first send a query broadcast or multicast message; it can also respond to the broadcast or multicast message after receiving the query broadcast or multicast message from the VR headset device in the network. After completing the discovery function, it can actively or passively establish a connection with the VR headset device.

[0189] After establishing a connection, as an example, the controlled device can act as an HTTP server and respond to the HTTP requests of the VR headset device in a RESTful manner. Of course, those skilled in the art should understand that the controlled device can also be set as an HTTP client and the VR headset device as an HTTP server. Any other network protocol can also be used, for example, defining a dedicated communication message format over a TCP / IP-based network protocol.

[0190] 2) Send subscription gesture information.

[0191] As an example, after establishing an HTTP connection, the VR headset device sends a GET request message to the controlled device to query the list of gesture information of the controlled device. The controlled device returns a response message and sends subscription gesture information to the VR headset device through the response message.

[0192] The subscription gesture information here includes a set of gesture description information, see the foregoing example.

[0193] In this step, the controlled device also sends the address and port for sending subsequent virtual reality images to the VR headset device. The VR headset device receives and saves this information. After sending the address and port of the VR image, the controlled device establishes a network connection with the VR headset device. Different network ports can be used for sending VR images and sending subscription gesture information.

[0194] 3) Send VR images.

[0195] The VR headset device establishes a network connection to the address and port for sending VR images. Here, different network ports can be used for receiving VR images and receiving subscription gesture information. As an example, the protocol for transmitting VR images can be a streaming media transmission protocol such as HLS, RSTP, DASH, or downloading files through HTTP.

[0196] The content of the VR image can be a panorama of the controlled device, or real scene photos from multiple angles, and can also be internal schematic photos of the controlled device, view images that change due to user manipulation, etc.

[0197] 4) Receive gesture information.

[0198] The HTTP server on the controlled device waits to receive messages from the VR headset device. When the VR headset device recognizes a gesture, it will send the recognized gesture information to the controlled device, for example, sending a message to the controlled device through an HTTP PUT message or an HTTP POST message, and the controlled device receives this gesture information.

[0199] 5) Judge according to the received gesture information. If it is necessary to update the virtual reality image, go to 3).

[0200] As an example, after the controlled device parses the received gesture information and finds that the display information to the user needs to be updated, such as replacing with a new device screen, it proceeds to step 3 and continues to send virtual reality images.

[0201] 6) And continue to analyze the received gesture information and execute the gesture control function.

[0202] If after the controlled device parses the received gesture information and determines that it needs to execute certain functions itself, such as a TV adjusting the volume, an air conditioner raising / lowering the temperature, turning on / off / dimming / brightening the lights, etc., it starts to execute the control function corresponding to the gesture.

[0203] 7) If the controlled device decides to end the gesture control process, it sends a message to the VR headset device to request the end of the process, otherwise it proceeds to step 4).

[0204] As an example, finally the controlled device closes the address and port connections with the VR headset device.

[0205] In the above embodiments, the TV set, player, and storage server are used as the controlled devices, but the present invention is not limited to the controlled devices being those mentioned in the embodiments. Computers, stereos, speakers, projectors, set-top boxes, etc. can all be used as controlled devices. Even other industrial devices such as cars, machine tools, ships, etc. can be controlled by the VR headset device.

[0206] In the above embodiments, the camera of the VR headset device can be of various specifications. For example, it can be a fixed focal length or variable focal length, and the rotation space can be at all angles of up, down, left, and right, or only support left and right angles. Only one camera needs to be configured to have various capabilities described in the embodiments. The ranging unit can use laser infrared ranging or measure with light of other wavelengths. It can use 3 cameras to calculate the ranging, or use more cameras and calculate the ranging using methods such as weighted adjustment.

[0207] The above process is an embodiment for implementing the present invention. It is not limited to being implemented only by the embodiments, nor is it used to limit the specific method of executing the process. The present invention can also be implemented in a similar manner. For example, the name of the representation unit, the types of various messages, etc. are just different in naming forms and specific message contents.

[0208] In the above embodiments, those related to networks are applicable to IP networks supported by communication networks such as IEEE 802.3, IEEE 802.11b / g / n, POWELINE, CABLE, PSTN, the 3rd Generation Partnership Project (3GPP) network, and 3GPP2 network. The operating systems of each device are applicable to UNIX-like operating systems, WINDOWS-like operating systems, ANDROID-like operating systems, and IOS operating systems. The consumer interfaces are applicable to JAVA language interfaces, etc.

[0209] Figure 10 Schematic diagram of the structural composition of the gesture recognition device according to an embodiment of the present invention Figure 1 , applied to a first device, such as Figure 10 As shown, the device includes:

[0210] A receiving unit 1000, configured to obtain a gesture list supported by a second device; receive VR data sent by the second device;

[0211] A display unit 1001, configured to display the VR image;

[0212] An acquisition and recognition unit 1002, configured to acquire and recognize gestures, and generate a gesture message based on the gesture list supported by the obtained second device, where the gesture message carries at least one of the following information: gesture information, control information;

[0213] A sending unit 1003, configured to send the gesture message to the second device.

[0214] In an embodiment of the present invention, the receiving unit 1000 is further configured to receive a subscribed gesture message sent by a second device, where the subscribed gesture message carries the gesture list supported by the second device.

[0215] In an embodiment of the present invention, the device further includes:

[0216] A connection unit 1004, configured to send a query broadcast or multicast message to perform device discovery on a second device; or, after the first device receives a query broadcast or multicast message sent by the second device, respond to the query broadcast or multicast message; after device discovery is completed between the first device and the second device, establish a network connection between the first device and the second device.

[0217] In an embodiment of the present invention, the connection unit 1004 is further configured to establish a network connection with the second device through the network address of the second device.

[0218] In an embodiment of the present invention, the gesture list includes at least one of the following: the gesture number supported by the second device, the gesture feature data corresponding to the gesture number, and the gesture control instruction corresponding to the gesture number.

[0219] In an embodiment of the present invention, the gesture information includes at least one of the following: gesture number, gesture parameter, current image parameter when the gesture is collected, position mapping parameter of the gesture on the current image, continuously collected gesture video stream, and predefined instruction code.

[0220] In an embodiment of the present invention, at least one of the following information is stored in the second device: VR data of the second electronic device, and the gesture list supported by the second device, where the VR data is VR image data or VR video data;

[0221] The subscription gesture message also carries the network address or network port corresponding to the VR data; the first device receives the VR data sent by the second device according to the network address or network port corresponding to the VR data.

[0222] Those skilled in the art should understand that Figure 10 The implementation functions of the various units in the gesture recognition device shown can be understood with reference to the relevant descriptions of the foregoing gesture recognition method.

[0223] The technical solution of the embodiment of the present invention enables the user to perform gesture control on the device without directly facing the controlled device, and it is also equally simple and convenient to control multiple devices, making people's entertainment life easier and more enjoyable. People do not need to learn and master more usage methods, and it can also reduce the production cost of enterprises and the consumption cost of users.

[0224] Figure 11 It is a schematic structural composition of the gesture recognition device according to an embodiment of the present invention Figure 2 , applied to a second device, such as Figure 11 As shown, the device includes:

[0225] A sending unit 1101, configured to send the gesture list supported by the second device to the first device; and send the VR data to the first device so that the first device displays the VR data;

[0226] A receiving unit 1102, configured to receive the gesture message sent by the first device, where the gesture message carries at least one of the following information: gesture information and control information;

[0227] An execution unit 1103, configured to execute a corresponding control instruction according to the gesture message.

[0228] In an embodiment of the present invention, the sending unit 1101 is further configured to send a subscription gesture message to a first device, where the subscription gesture message carries a gesture list supported by the second device.

[0229] In an embodiment of the present invention, the apparatus further includes:

[0230] A connection unit 1104, configured to send a query broadcast or multicast message to perform device discovery on the first device; alternatively, after the second device receives a query broadcast or multicast message sent by the first device, the second device responds to the query broadcast or multicast message; after the second device and the first device complete device discovery, the second device establishes a network connection with the first device.

[0231] In an embodiment of the present invention, the connection unit 1104 is further configured to establish a network connection with the first device through the network address of the first device.

[0232] In an embodiment of the present invention, the gesture list includes at least one of the following: a gesture number supported by the second device, gesture feature data corresponding to the gesture number, and a gesture control instruction corresponding to the gesture number.

[0233] In an embodiment of the present invention, the gesture information includes at least one of the following: a gesture number, gesture parameters, current image parameters when the gesture is collected, position mapping parameters of the gesture on the current image, a continuously collected gesture video stream, and a predefined instruction code.

[0234] In an embodiment of the present invention, the second device stores at least one of the following information: VR data of the second electronic device, a gesture list supported by the second device, where the VR data is VR image data or VR video data;

[0235] The subscription gesture message further carries a network address or network port corresponding to the VR data.

[0236] Those skilled in the art should understand that Figure 11 The implementation functions of the units in the gesture recognition device shown can be understood with reference to the relevant descriptions of the foregoing gesture recognition method.

[0237] Figure 12 It is a schematic structural composition diagram of a gesture recognition system according to an embodiment of the present invention. The system includes: a first device 1201 and a second device 1202; where

[0238] The first device 1201 is configured to obtain a list of gestures supported by the second device 1202; receive VR data sent by the second device 1202 and display the VR data; collect and recognize gestures, generate a gesture message based on the obtained list of gestures supported by the second device 1202, where the gesture message carries at least one of the following information: gesture information, control information; and send the gesture message to the second device 1202.

[0239] The second device 1202 is configured to send the list of gestures supported by the second device 1202 to the first device 1201; send VR data to the first device 1201 so that the first device 1201 can display the VR data; receive the gesture message sent by the first device 1201, where the gesture message carries at least one of the following information: gesture information, control information; and execute corresponding control instructions according to the gesture message.

[0240] In an embodiment of the present invention, the list of gestures includes at least one of the following: the gesture numbers supported by the second device 1202, the gesture feature data corresponding to the gesture numbers, and the gesture control instructions corresponding to the gesture numbers.

[0241] In an embodiment of the present invention, the gesture information includes at least one of the following: gesture numbers, gesture parameters, current image parameters when the gesture is collected, position mapping parameters of the gesture on the current image, continuously collected gesture video streams, and predefined instruction codes.

[0242] For a device without a gesture recognition component (such as a device without a camera) in the technical solution of the embodiment of the present invention, the user does not need to face the device, and can also control the device (such as selecting functions, configuring parameters, etc.) through gestures, and the cost increase of the device itself is relatively low.

[0243] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of an embodiment implemented in hardware, an embodiment implemented in software, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0244] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0245] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0246] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0247] As described above, it is only the preferred embodiment of the present invention, and is not used to limit the protection scope of the present invention.

Claims

1. A gesture recognition method, characterized in that, The method includes: The first device obtains the gesture list supported by the second device; The first device receives the virtual reality (VR) data sent by the second device and displays the VR data, where the VR data is used to provide a visual reference facilitating gesture operation corresponding to the second device; The first device collects and recognizes gestures, generates a gesture message based on the obtained gesture list supported by the second device, and the gesture message carries at least one of the following information: gesture information, control information, where the gesture information includes the current image parameters when the gesture is collected, the position mapping parameters of the gesture on the current image, and a predefined instruction code, and the instruction code is mapped after being predefined by the first device for the gesture and is subscribed in advance by the second device; The first device sends the gesture message to the second device, so that the second device updates the VR data according to the current image parameters when the gesture is collected and the position mapping parameters of the gesture on the current image and sends it to the first device.

2. The gesture recognition method according to claim 1, wherein The first device obtains the gesture list supported by the second device, including: The first device receives the subscribed gesture message sent by the second device, and the subscribed gesture message carries the gesture list supported by the second device.

3. The gesture recognition method according to claim 2, wherein Before the first device receives the subscribed gesture message sent by the second device, the method further includes: The first device sends a query broadcast or multicast message to discover the second device; or, after the first device receives the query broadcast or multicast message sent by the second device, it responds to the query broadcast or multicast message; After the first device and the second device complete device discovery, the first device and the second device establish a network connection.

4. The gesture recognition method according to claim 2, characterized in that, Before the first device receives the subscribed gesture message sent by the second device, the method further includes: The first device establishes a network connection with the second device through the network address of the second device.

5. The gesture recognition method according to claim 1, wherein The gesture list includes at least one of the following: the gesture numbers supported by the second device, the gesture feature data corresponding to the gesture numbers, and the gesture control instructions corresponding to the gesture numbers.

6. The gesture recognition method according to claim 1, wherein The gesture information further includes at least one of the following: gesture numbers, gesture parameters, and continuously collected gesture video streams.

7. The gesture recognition method according to claim 1, characterized in that, The second device stores at least one of the following information: the VR data of the second electronic device, the gesture list supported by the second device, where the VR data is VR image data or VR video data; The subscribed gesture message also carries the network address or network port corresponding to the VR data; the first device receives the VR data sent by the second device according to the network address or network port corresponding to the VR data.

8. A gesture recognition method, characterized in that The method includes: The second device sends the gesture list supported by the second device to the first device; The second device sends the VR data to the first device, so that the first device displays the VR data, and the VR data is used to provide a visual reference facilitating gesture operation corresponding to the second device; The second device receives the gesture message sent by the first device, and the gesture message carries at least one of the following information: gesture information, control information, where the gesture information includes the current image parameters when the gesture is collected, the position mapping parameters of the gesture on the current image, and a predefined instruction code, and the instruction code is mapped after being predefined by the first device for the gesture and is subscribed in advance by the second device; Execute the corresponding control instruction according to the gesture message, and at the same time update the VR data according to the current image parameters when the gesture is collected and the position mapping parameters of the gesture on the current image; Send the updated VR data to the first device for display.

9. The gesture recognition method according to claim 8, wherein The second device sends a gesture list supported by the second device to the first device, including: The second device sends a subscription gesture message to the first device, and the subscription gesture message carries the gesture list supported by the second device.

10. The gesture recognition method according to claim 9, characterized in that The second device sends to Before the second device sends a subscription gesture message to the first device, the method further includes: The second device sends a query broadcast or multicast message to discover the first device; or, after the second device receives the query broadcast or multicast message sent by the first device, it responds to the query broadcast or multicast message; After the second device and the first device complete device discovery, the second device establishes a network connection with the first device.

11. The gesture recognition method according to claim 9, wherein Before the second device sends a subscription gesture message to the first device, the method further includes: The second device establishes a network connection with the first device through the network address of the first device.

12. The gesture recognition method according to claim 9, wherein The gesture list includes at least one of the following: the gesture numbers supported by the second device, the gesture feature data corresponding to the gesture numbers, and the gesture control instructions corresponding to the gesture numbers.

13. The gesture recognition method according to claim 9, wherein The gesture information further includes at least one of the following: gesture numbers, gesture parameters, and continuously collected gesture video streams.

14. The gesture recognition method according to claim 9, wherein, At least one of the following information is stored in the second device: the VR data of the second electronic device, the gesture list supported by the second device, where the VR data is VR image data or VR video data; The subscription gesture message also carries the network address or network port corresponding to the VR data.

15. A gesture recognition device, characterized in that, Applied to the first device, the device includes: A receiving unit, configured to obtain the gesture list supported by the second device; receive the VR data sent by the second device, where the VR data is used to provide a visual reference convenient for gesture operation corresponding to the second device; A display unit, configured to display the VR data; An acquisition and recognition unit, configured to acquire and recognize gestures, and generate a gesture message based on the obtained gesture list supported by the second device, where the gesture message carries at least one of the following information: gesture information, control information, where the gesture information includes the current image parameters when the gesture is collected, the position mapping parameters of the gesture on the current image, and a predefined instruction code, and the instruction code is mapped after being predefined by the first device for the gesture and is subscribed in advance by the second device; A sending unit, configured to send the gesture message to the second device, so that the second device updates the VR data according to the current image parameters when the gesture is collected and the position mapping parameters of the gesture on the current image, and sends the updated VR data to the first device.

16. The gesture recognition device according to claim 15, wherein The receiving unit is further configured to receive a subscribed gesture message sent by the second device, where the subscribed gesture message carries a list of gestures supported by the second device.

17. The gesture recognition device according to claim 16, wherein The device further includes: A connection unit, configured to query a broadcast or multicast message to perform device discovery on the second device; or, after the first device receives a query broadcast or multicast message sent by the second device, respond to the query broadcast or multicast message; after the first device and the second device complete device discovery, the first device and the second device establish a network connection.

18. The gesture recognition device according to claim 16, wherein The device further includes: A connection unit, configured to establish a network connection with the second device through the network address of the second device.

19. The gesture recognition device according to claim 15, characterized in that, The list of gestures includes at least one of the following: the gesture numbers supported by the second device, the gesture feature data corresponding to the gesture numbers, and the gesture control instructions corresponding to the gesture numbers.

20. The gesture recognition device according to claim 15, wherein, The gesture information further includes at least one of the following: gesture numbers, gesture parameters, and continuously collected gesture video streams.

21. The gesture recognition device according to claim 15, characterized in that The second device stores at least one of the following information: the VR data of the second electronic device, the list of gestures supported by the second device, where the VR data is VR image data or VR video data; The subscribed gesture message further carries the network address or network port corresponding to the VR data; the first device receives the VR data sent by the second device according to the network address or network port corresponding to the VR data.

22. A gesture recognition device, characterized in that, Applied to the second device, the device includes: A sending unit, configured to send the list of gestures supported by the second device to the first device; send the VR data to the first device, so that the first device displays the VR data, and the VR data is used to provide a visual reference convenient for gesture operation corresponding to the second device; A receiving unit, configured to receive the gesture message sent by the first device, where the gesture message carries at least one of the following information: gesture information, control information, where the gesture information includes the current image parameters when the gesture is collected, the position mapping parameters of the gesture on the current image, and a predefined instruction code, and the instruction code is mapped after being predefined by the first device for the gesture and is subscribed in advance by the second device; An execution unit, configured to execute corresponding control instructions according to the gesture message, and at the same time update the VR data according to the current image parameters when the gesture is collected and the position mapping parameters of the gesture on the current image; Send the updated VR data to the first device for display.

23. The gesture recognition device according to claim 22, characterized in that, The sending unit is further configured to send a subscribed gesture message to the first device, where the subscribed gesture message carries the list of gestures supported by the second device.

24. The gesture recognition device according to claim 23, wherein The device further includes: A connection unit for sending a query broadcast or multicast message to perform device discovery on a first device; alternatively, after the second device receives a query broadcast or multicast message sent by the first device, the second device responds to the query broadcast or multicast message; after the second device and the first device complete device discovery, the second device establishes a network connection with the first device.

25. The gesture recognition device according to claim 23, wherein The device further includes: A connection unit for establishing a network connection with the first device through the network address of the first device.

26. The gesture recognition device according to claim 22, wherein The gesture list includes at least one of the following: the gesture number supported by the second device, the gesture feature data corresponding to the gesture number, and the gesture control instruction corresponding to the gesture number.

27. The gesture recognition device according to claim 22, wherein The gesture information includes at least one of the following: gesture number, gesture parameter, current image parameter when the gesture is collected, position mapping parameter of the gesture on the current image, continuously collected gesture video stream, and predefined instruction code.

28. The gesture recognition device according to claim 22, wherein The second device stores at least one of the following information: VR data of the second electronic device, gesture list supported by the second device, where the VR data is VR image data or VR video data; The subscribed gesture message also carries the network address or network port corresponding to the VR data.

29. A gesture recognition system, characterized in that, The system includes: a first device, a second device; where The first device is configured to obtain the gesture list supported by the second device; receive the VR data sent by the second device and display the VR data, where the VR data is used to provide a visual reference for gesture operation corresponding to the second device; collect and recognize gestures, generate a gesture message based on the obtained gesture list supported by the second device, where the gesture message carries at least one of the following information: gesture information, control information, where the gesture information includes the current image parameter when the gesture is collected, the position mapping parameter of the gesture on the current image, and a predefined instruction code, and the instruction code is mapped after being predefined by the first device for the gesture and is subscribed in advance by the second device; send the gesture message to the second device, so that the second device updates the VR data according to the current image parameter when the gesture is collected and the position mapping parameter of the gesture on the current image and sends it to the first device; The second device is configured to send a list of gestures supported by the second device to the first device; send VR data to the first device so that the first device can display the VR data, where the VR data is used to provide a visual reference for gesture operation corresponding to the second device; receive a gesture message sent by the first device, where the gesture message carries at least one of the following information: gesture information and control information, where the gesture information includes the current image parameters when the gesture is collected, the position mapping parameters of the gesture on the current image, and a predefined instruction code, and the instruction code is mapped after being predefined by the first device for the gesture and is subscribed in advance by the second device; execute corresponding control instructions according to the gesture message, and at the same time update the VR data according to the current image parameters when the gesture is collected and the position mapping parameters of the gesture on the current image, and send the updated VR data to the first device for display.

30. The gesture recognition system according to claim 29, wherein The gesture list includes at least one of the following: the gesture numbers supported by the second device, the gesture feature data corresponding to the gesture numbers, and the gesture control instructions corresponding to the gesture numbers.

31. The gesture recognition system according to claim 29, wherein The gesture information further includes at least one of the following: gesture numbers, gesture parameters, and continuously collected gesture video streams.

Images