Device control method, electronic device and computer program product

Abstract

The invention discloses an equipment control method, electronic equipment and a computer program product, and belongs to the technical field of information processing. The equipment control method in the embodiment of the invention comprises the following steps: acquiring a screen video stream transmitted by terminal equipment; displaying the screen video stream through a virtual reality VR device; according to an input operation executed by a user based on the screen video stream, obtaining input information and / or an operation instruction; and transmitting the input information and / or the operation instruction to the terminal equipment, and executing corresponding operation by the terminal equipment according to the input information and / or the operation instruction. Therefore, the VR experience of the user can be improved.

Description

Technical Field

[0001] The present application belongs to the field of information processing technology, and specifically relates to a device control method, an electronic device, and a computer program product. Background Art

[0002] In the related technologies, most virtual reality (VR) devices usually only perform related displays and operations based on their own situations, and a few VR devices can realize screen projection display of pages received from terminal devices (such as mobile phones). As a result, VR devices and other terminal devices (such as mobile phones) are almost isolated, and the two cannot be integrated. For example, when a user wants to operate a mobile phone while wearing a VR device, he usually needs to take off the VR device to operate the physical mobile phone, or cancel the VR-related functions and operate the physical mobile phone through glasses, which will interrupt the user's VR experience and make the user's VR experience poor. In this case, how to improve the user's VR experience is an urgent problem to be solved. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a device control method, an electronic device, and a computer program product to solve the problem of how to enhance the user's VR experience.

[0004] In order to solve the above technical problems, this application is implemented as follows:

[0005] In a first aspect, a device control method is provided, which is applied to a VR device, comprising:

[0006] Get the screen video stream transmitted by the terminal device;

[0007] Displaying the screen video stream through a virtual reality VR device;

[0008] Acquiring input information and / or operation instructions according to an input operation performed by a user based on the screen video stream;

[0009] The input information and / or operation instruction is transmitted to the terminal device, and the terminal device performs corresponding operations according to the input information and / or operation instruction.

[0010] In a second aspect, a device control method is provided, which is applied to a terminal device, comprising:

[0011] The collected screen video stream is transmitted to a VR device, and the VR device displays the screen video stream;

[0012] Acquire input information and / or operation instructions transmitted by the VR device; wherein the input information and / or operation instructions are obtained according to input operations performed by the user based on the screen video stream;

[0013] Execute corresponding operations according to the input information and / or operation instructions.

[0014] In a third aspect, a device control device is provided, which is applied to a VR device, including:

[0015] A first acquisition module, used to acquire a screen video stream transmitted by a terminal device;

[0016] A display module, used to display the screen video stream through a VR device;

[0017] A second acquisition module, used for acquiring input information and / or operation instructions according to an input operation performed by a user based on the screen video stream;

[0018] The first transmission module is used to transmit the input information and / or operation instruction to the terminal device, and the terminal device performs corresponding operations according to the input information and / or operation instruction.

[0019] In a fourth aspect, a device control apparatus is provided, which is applied to a terminal device, including:

[0020] A fourth transmission module is used to transmit the collected screen video stream to the VR device, and the VR device displays the screen video stream;

[0021] A third acquisition module is used to acquire input information and / or operation instructions transmitted by the VR device; wherein the input information and / or operation instructions are obtained according to the input operation performed by the user based on the screen video stream;

[0022] The execution module is used to execute corresponding operations according to the input information and / or operation instructions.

[0023] In a fifth aspect, an electronic device is provided, comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect, or the steps of the method described in the second aspect.

[0024] In a sixth aspect, a readable storage medium is provided, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect or the steps of the method described in the second aspect are implemented.

[0025] In a seventh aspect, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the steps of the method described in the first aspect or the steps of the method described in the second aspect.

[0026] Through the scheme of the embodiment of the present application, after obtaining the screen video stream transmitted by the terminal device, the VR device can display the screen video stream, and according to the input operation performed by the user based on the screen video stream, obtain input information and / or operation instructions, transmit the input information and / or operation instructions to the terminal device, and the terminal device performs corresponding operations according to the input information and / or operation instructions. In this way, the terminal device (such as a mobile phone) can be controlled by the VR device, so that the user can also control other terminal devices while using the VR device, without removing the VR device or canceling VR-related functions, so as to achieve the effect of operating and using other terminal devices in the VR environment, thereby avoiding interruption of the user's VR experience and improving the user's VR experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a flow chart of a device control method provided by an embodiment of the present application;

[0028] Figure 2 is a schematic diagram of key parts of the hand in an embodiment of the present application;

[0029] Figure 3A , Figure 3B , Figure 3C and Figure 3D is a schematic diagram of different hand gestures in the embodiments of the present application;

[0030] Figure 4A , Figure 4B , Figure 4C and Figure 4D is a schematic diagram of motion trajectories under different gesture postures in an embodiment of the present application;

[0031] Figure 5 is a schematic diagram of gesture operation positions in an embodiment of the present application;

[0032] Figure 6 is a flow chart of another device control method provided by an embodiment of the present application;

[0033] Figure 7 This is a schematic diagram of the interaction between the VR device and the mobile phone in the embodiment of the present application;

[0034] Figure 8 It is a structural schematic diagram of a device control device provided in an embodiment of the present application;

[0035] Fig. 9 is a structural schematic diagram of another device control device provided in an embodiment of the present application;

[0036] Fig.10 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0038] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0039] The device control method, electronic device and computer program product provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0040] See also Figure 1 , Figure 1 is a flow chart of a device control method provided in an embodiment of the present application, which is applied to VR devices, such as Figure 1 As shown, the method comprises the following steps:

[0041] Step 11: Get the screen video stream transmitted by the terminal device.

[0042] In the embodiment of the present application, the terminal device may be selected as but not limited to a mobile phone, a watch, a tablet computer, etc. The terminal device may interact with the VR device through the client APP therein. For example, during the initial connection, the relevant APP in the terminal device may scan the nearby selectable VR devices based on the functional interface provided by the operating system, such as the WifiP2pManager of Android or the MultipeerConnectivity interface of iOS, display the creation connection interface in the application, list the scanned connectable VR devices, and establish the connection between the terminal device and the VR device based on the user's selection.

[0043] The screen video stream may be obtained by recording, such as recording the real-time screen content and audio content of the terminal device. For example, after starting recording, the terminal device may use the MediaProjection API of the Android system or the ReplayKit of the iOS system to record the screen content in real time, and / or use the AudioRecord of the Android system or the AVAudioRecorder of the iOS system to record the audio in real time.

[0044] Optionally, in order to reduce the amount of data transmission, the terminal device (such as a mobile phone) can use a video encoder (such as H.264 or High Efficiency Video Coding HEVC) to compress the collected video data and audio data, and use an audio encoder (such as Opus or AAC) to compress the audio data to obtain compressed audio and video data (i.e., screen video stream) and transmit it to the VR device.

[0045] In an optional implementation, the screen video stream can be transmitted from the terminal device to the VR device via a Socket connection between the terminal device and the VR device to ensure the stability and real-time performance of the transmitted data and to control the delay within an acceptable range, such as controlling the delay within tens of milliseconds.

[0046] Step 12: Display the screen video stream through a VR device.

[0047] In an embodiment of the present application, after obtaining the screen video stream transmitted by the terminal device, the VR device can first decode the obtained screen video stream, and then render it in a VR environment based on the decoded audio and video data to display the real-time screen content and audio content of the terminal device, that is, to display the real-time screen image and audio of the terminal device.

[0048] The presentation / display position of the screen video stream may be set on the VR device.

[0049] It should be pointed out that when displaying screen video streams, VR devices can process the display content accordingly based on their own characteristics. For example: appropriately project the screen content of the terminal device to adapt to the field of view of the human eye in the VR device, so that the display content is more in line with the user's eye observation position; according to the lens characteristics of the VR device, the video frame is appropriately optimized and the distortion is corrected to eliminate the image deformation caused by the lens; the video frame is set with a suitable dynamic refresh frequency and frame rate to ensure the smoothness and stability of the content; and so on.

[0050] Step 13: Obtain input information and / or operation instructions according to the input operation performed by the user based on the screen video stream.

[0051] Optionally, the input operation may be a gesture operation or a voice input operation, which is not limited. The input information may be voice information, text information, etc. The operation instruction may be generated according to a preset instruction format, for example, the user voice "open APP1" may be identified as two instructions "open" and "APP1".

[0052] Step 14: Transmit the input information and / or operation instruction to the terminal device, and the terminal device performs corresponding operations according to the input information and / or operation instruction.

[0053] Optionally, the VR device may transmit the input information and / or operation instructions to the terminal device via a Socket connection between the VR device and the terminal device.

[0054] The corresponding operations performed by the terminal device may include but are not limited to making or receiving calls, opening an APP, operating the screen, playing a video, pausing a video, dragging and fast-forwarding a video, and the like.

[0055] In an optional implementation, after the terminal device executes the corresponding operation, the VR device can display the execution result to the user through user interface (UI) feedback or voice feedback, etc. For example, after the terminal device opens the APP, the application interface is displayed through the VR device, or after the terminal device sends a text message, the text message sending status is notified through voice, etc.

[0056] Through the scheme of the embodiment of the present application, after obtaining the screen video stream transmitted by the terminal device, the VR device can display the screen video stream, and according to the input operation performed by the user based on the screen video stream, obtain input information and / or operation instructions, transmit the input information and / or operation instructions to the terminal device, and the terminal device performs corresponding operations according to the input information and / or operation instructions. In this way, the terminal device (such as a mobile phone) can be controlled by the VR device, so that the user can also control other terminal devices while using the VR device, without removing the VR device or canceling VR-related functions, so as to achieve the effect of operating and using other terminal devices in the VR environment, thereby avoiding interruption of the user's VR experience, achieving the continuity and consistency of the VR experience to the greatest extent, and improving the user's VR experience.

[0057] Optionally, the solution in this application can be applied to VR headsets, VR glasses, somatosensory devices, etc.

[0058] In the embodiment of the present application, the VR device mainly establishes a two-way transmission channel between itself and a terminal device (such as a mobile phone), displays the acquired screen video stream of the terminal device in the user's current VR field of view, and can also adjust the display position, size, etc. of the terminal screen video according to user instructions, identify and parse user-specific control gestures and / or voice commands, and send user operation instructions to the terminal device to achieve the effect of remote control of the terminal device; in addition, the VR device can set a buffer for receiving video and audio data to ensure stable two-way communication; the VR device can also use the two-way transmission channel between itself and the terminal device to synchronize the pictures, audio, video and other content currently displayed by the VR device to the terminal device, and the terminal device will perform subsequent specific processing.

[0059] Optionally, before obtaining the screen video stream transmitted by the terminal device, the device control method in the embodiment of the present application may further include:

[0060] In response to the link request of the terminal device, a bidirectional transmission channel is established between the VR device and the terminal device; that is, according to this bidirectional transmission channel, the VR device can transmit instructions / information to the terminal device, and the terminal device can also transmit instructions / information to the VR device. The above link request can be triggered based on user operation. For example, after the user selects the VR device to be connected, the terminal device sends a link request to the selected VR device to establish a bidirectional transmission channel between the VR device and the terminal device.

[0061] The above-mentioned obtaining the screen video stream transmitted by the terminal device may include: the VR device obtains the screen video stream transmitted by the terminal device through the bidirectional transmission channel. In this way, with the help of the bidirectional transmission channel, stable bidirectional communication between the VR device and the terminal device can be ensured.

[0062] Optionally, the device control method in the embodiment of the present application may further include:

[0063] In response to the user's input operation, the display content of the VR device is transmitted to the terminal device; wherein the input operation can be a gesture operation or a voice input operation, and the display content can be, for example, pictures, audio, video, etc.;

[0064] Generate a control instruction according to a control operation input by a user based on the displayed content; for example, the control operation may be a gesture operation or a voice input operation, and the control instruction may be generated according to a preset instruction format;

[0065] The control instruction is transmitted to the terminal device, and the terminal device processes the display content according to the control instruction.

[0066] In this way, for terminal devices that need to read external images in reverse, real-time video transmission and reading from VR to terminal devices can be achieved; operations that usually require cooperation between VR devices and terminal devices can be changed to operations that only need to be performed on the VR side, without the need to operate multiple devices at the same time, thereby achieving the effect of controlling the terminal device through the VR device to process the images, audio and video displayed by the VR device itself, realizing the capability integration and joint control of VR devices and terminal devices, and improving user experience and operating efficiency.

[0067] Optionally, for the scenario of synchronizing the audio and video of the VR device to the terminal device (such as a mobile phone) for subsequent processing, according to the specific processing type, it can be divided into two categories: asynchronous processing and real-time synchronous processing; among them, asynchronous processing is suitable for scenarios that do not require real-time reading operations, such as the more common user scanning code login, scanning code payment and other scenarios; but in actual applications, there are scenarios that require real-time synchronous dynamic reading of videos or images, in which case the terminal device needs to adopt real-time synchronous processing. The specific implementation process and detailed implementation instructions of these two processing methods are as follows.

[0068] (1) Asynchronous processing:

[0069] In an asynchronous processing mode, the VR device can transmit the image (such as containing a QR code) displayed on the current VR screen to a terminal device such as a mobile phone; the mobile phone can first save the image to local storage (such as mobile phone storage), and then after receiving the control instruction sent by the VR device (the control instruction is generated based on the user's input operation in the VR environment), the VR device controls the mobile phone to process the stored image based on the control instruction. For example, if the control instruction is to open application 1 to scan the QR code in the image for login or payment, then open application 1 and select the image to be scanned to scan the code for login or scan the code for payment; or, if the control instruction is to adjust the background color of the image, then adjust the background color of the stored image.

[0070] In addition, video, audio, etc. can also be applied to asynchronous processing. For example, the VR device transmits the currently displayed audio and video to the mobile phone (or mobile phone APP); the mobile phone can first save the audio and video to the local storage, and then after receiving the control instruction sent by the VR device (the control instruction is generated based on the user's input operation in the VR environment), the VR device controls the mobile phone and processes the audio and video stored in the mobile phone based on the control instruction, such as adjusting the frame rate, clarity, etc. The process and results of the mobile phone processing can be displayed synchronously on the VR screen in real time.

[0071] (2) Real-time synchronous processing:

[0072] In the real-time synchronous processing mode, the terminal device can obtain and process the video / image content synchronized with the VR device by reading the camera (such as a virtual camera) in real time.

[0073] Taking the mobile phone as an example, the VR device can synchronously transmit the currently displayed audio and video to the mobile phone APP; at the same time, the mobile phone APP saves the real-time audio and video as a local file, creates a virtual camera, and then reads the saved local file, decodes the saved image into a real-time data stream and outputs it to the virtual camera, and then passes it to the application that needs to read the camera in real time through the virtual camera, so as to realize the transmission of the audio and video of the VR device through the real-time data stream. For applications with high real-time requirements, the format of the audio and video provided by the VR device can be directly converted into the format required by the mobile phone camera in real time, and then provided to the virtual camera, and then provided to the corresponding mobile phone application. At this time, open the application in the mobile phone in VR, and the audio and video synchronized with the VR device can be processed by the mobile phone, so as to realize the real-time transmission and subsequent processing of audio and video from the VR device to the mobile phone application. The process and results of the mobile phone processing are displayed on the VR screen in real time.

[0074] It should be noted that whether to use an asynchronous processing method or a real-time synchronous processing method can be determined by the terminal device. For example, after the VR device synchronizes its current display content to the terminal device, or after the terminal device reads the current display content of the VR device, the terminal device can select a processing method according to the processing requirements and / or the received instructions. For example, if it is not a scene that requires real-time processing, then the asynchronous processing method is selected, otherwise the real-time synchronous processing method is selected.

[0075] Since the human hand has many ways of moving, which are complex and difficult to summarize, if the VR device synchronizes every tiny movement of the finger to the terminal device (such as a mobile phone) without reservation, in addition to causing information redundancy, it will also increase the amount of information processing of both devices, increase memory and system overhead. In addition, the user's hand is the most core operating part, and the movements of the human arm, leg and other parts are meaningless for operating the mobile phone. Therefore, the operation instructions can be generated based on the recognition of the user's hand.

[0076] Considering that the common operations of operating a mobile phone can be divided into three categories: single-finger operation, two-finger operation, and overall operation; among them, single-finger operation can be further divided into click, slide (such as up, down, left, and right) and long press; two-finger operation can be further divided into zoom in and zoom out; overall operation is the overall movement of the hand, and the corresponding function can be divided into mobile phone flipping and horizontal and vertical screen switching. Therefore, user gesture operations can be identified based on the user's gesture posture and hand movement trajectory, and then operation instructions can be generated.

[0077] Optionally, if the user's input operation is a gesture operation, the process of obtaining the operation instruction in the above step 13 may include:

[0078] Recognize the gesture operation to obtain the user's gesture posture and movement trajectory information of key parts of the user's hand under the gesture posture;

[0079] The operation instruction is generated according to the gesture posture and the motion trajectory information.

[0080] The above recognition of gesture operations can be understood as: recognizing the movement of the user's hand and extracting the core hand movements of the user's operation to convert them into action instructions, and passing the action instructions to the terminal device for subsequent processing. In this way, the user's gesture operation can be accurately recognized based on the user's gesture posture and hand movement trajectory, so as to accurately generate operation instructions and cooperate to achieve the effect of remote control of the terminal device (such as a mobile phone).

[0081] It should be pointed out that in order to recognize gesture operations, the VR device can collect the user's hand movements through a depth camera, and obtain the user's gesture posture and hand movement trajectory based on the collected information.

[0082] In reality, mobile phone operations are all based on the screen, which can be understood as an operation method that is mainly based on a two-dimensional plane, supplemented by a small amount of three-dimensional movement. In VR, the user's hand movements are all three-dimensional, so the user's three-dimensional movement recognition needs to be converted into two-dimensional operations.

[0083] The user's hand motion trajectory can be characterized by defining the motion trajectory of the key parts of the user's hand. Figure 2 As shown, the key parts of the user's hand include at least but are not limited to the three nodes of the thumb, the joints of the other four fingers, the fingertip positions and the relative positions with the thumb, the palm (back of hand) area, etc., among which the thumb tip and the index finger tip are the core nodes. Based on these key parts, the accurate position of the user's hand, gesture posture and hand movement trajectory can be determined.

[0084] Optionally, the gesture posture may include but is not limited to a single-finger gesture (such as Figure 3A ), two-finger gestures (such as Figure 3B as shown), overall gesture posture (such as Figure 3C ) and flip gesture poses (as shown Figure 3D shown).

[0085] In an optional implementation, the user's hand gesture may be determined based on the relative positions of the key parts of the user's hand that are identified. Figure 3AIf the relative positions of the key parts of the hand are similar (such as the position of the index finger tip is more prominent than the positions of other finger tips), the user's gesture posture can be determined to be a single-finger gesture; if the relative positions of the key parts of the hand identified are similar to Figure 3B If the relative positions of the key parts of the hand are similar (such as the fingertips of the index finger and thumb are more prominent), the user's gesture posture can be determined to be a two-finger gesture; if the relative positions of the key parts of the hand are similar to Figure 3C If the relative positions of the key parts in the hand are similar, the user's gesture posture can be determined as an overall gesture posture; if the relative positions of the key parts of the user's hand identified are similar to Figure 3D If the relative positions of the key parts in the figure are similar, it can be determined that the user's gesture is a flip gesture.

[0086] When confirming the current accurate gesture posture, the three-dimensional spatial coordinate values ​​of the key parts of the current finger can be identified at the same time (generally taking the center of the field of view as the reference origin), that is, the values ​​of the three-dimensional coordinate system of X (left and right, positive to the right), Y (up and down, positive to the upward), and Z (front and back, positive to the forward) as the initial values ​​for subsequent processing.

[0087] Considering the timeliness of the user's actions in operating the device, when identifying the user's gesture operation through the video captured by the depth camera, the video processing time should not be too long, generally controlled within 0.5S, and the hand movement of this length is used as a minimum processing unit. At this time, by processing the video image (it can be combined with I-frame and P-frame video images for processing to improve the processing speed), the movement direction of the user's gesture in the two-dimensional plane (such as up / down / left / right) is obtained; at the same time, based on the depth image, the coordinate value of the user's gesture after the movement is completed can be obtained, and combined with the coordinate changes of the key points of the user's hand, the user's hand movement trajectory can be identified. Then, through the identified user gesture posture and user hand movement trajectory, the action most similar to the standard action can be determined, and the relevant action instructions can be matched.

[0088] Taking into account the characteristics of actual hand movements, a specific action cannot guarantee that only one-dimensional coordinates change. It is very likely that two or even three coordinates change at the same time. Therefore, different three-dimensional coordinate change differences need to be considered for different actions.

[0089] The following is an explanation of the gesture operation recognition method in the embodiment of the present application.

[0090] 1. Single Finger Gesture

[0091] When it is confirmed that the user's initial gesture is a single-finger gesture, it is only necessary to process the coordinates of the key part of the index finger tip and compare them with the original coordinate position, that is, to identify the change difference of the three-dimensional coordinates X, Y, and Z of the index finger tip.

[0092] Sliding (up, down, left, and right): The change in the Z axis is small, while the change in the X or Y axis is large. The largest change represents the sliding direction. At the same time, the sliding speed can be matched according to the speed of the index finger tip to respond to the user's fast sliding action. In order to match the sliding distance of the screen, it is necessary to calculate the sliding distance and speed of the index finger. By obtaining the coordinates of the key parts of the index finger tip, the distance change of the index finger tip in the X and Y axis planes is calculated. The calculation formula is, for example: X-axis coordinate difference 2 +Y axis coordinate difference 2 =Distance 2 , where the coordinate difference is for the index fingertip, the movement distance of the index fingertip before and after the user's action can be obtained; the time taken for the key part of the user's index fingertip to complete the movement is used as the time parameter t, and the distance / time t=speed v can be used to obtain the speed of the user's index finger sliding, and then the user's sliding type can be distinguished by the size of the speed, such as normal sliding, fast sliding, etc. At this time, the identification of key parts and movement recognition can be as follows Figure 4A shown.

[0093] Click: The changes in the X-axis and Y-axis are relatively small, while the change in the Z-axis is the largest. After the action is completed, the user does not maintain the original posture. In this case, the user operation is identified as a click operation. For click actions, there is no need to calculate the absolute distance and speed of the user's finger movement.

[0094] Long press: The changes in the X-axis and Y-axis are relatively small, while the change in the Z-axis is the largest. After the action is completed, the user action remains paused for a certain period of time. At this time, the user operation is recognized as a long press operation. For long press actions, there is no need to calculate the absolute distance and speed of the user's finger movement.

[0095] 2. Two-finger gesture

[0096] When it is confirmed that the user's initial gesture is a two-finger gesture, since the two-finger zoom-in and zoom-out actions are essentially a gesture, at this time, it is only necessary to identify the coordinates of the key parts of the thumb and index finger tips, and calculate the distance change between the thumb and index finger tips in the X-axis and Y-axis planes. The calculation formula is, for example: X-axis coordinate difference 2 +Y axis coordinate difference 2 =Distance 2 (The coordinate difference here is the coordinate difference between the thumb and the index finger, which is different from the calculation method of the sliding type mentioned above), the change in the distance between the thumb and the index finger before and after the user's action can be obtained, and the user's zoom in and zoom out operation can be matched. At this time, the key parts and motion recognition can be Figure 4B shown.

[0097] 3. Overall hand gestures and postures

[0098] When it is confirmed that the user's initial gesture is an overall gesture, the movement trajectory of the joints at the base of the fingers can be determined. At this time, the changes in the coordinate position of the user's fingers are ignored, and the movement directions of the four joints at the base of the fingers are calculated at the same time, that is, the distance changes of the four joints on the Y axis are calculated to obtain the changes in the four joints in the Y-axis direction before and after the user's action. For example, the average value can be used for calculation; when the movement direction is upward, it is recognized as switching from vertical screen to horizontal screen, and vice versa, it is an operation of switching from horizontal screen to vertical screen. At this time, the key parts of recognition and movement recognition can be as follows: Figure 4C As shown:

[0099] 4. Flip gesture posture

[0100] When it is confirmed that the user's initial gesture is a flip gesture, the movement trajectory of the thumb and index finger joints can be determined. The characteristics of this type of operation are that the Z-axis changes are small, the index finger's movement distance in the X and Y planes is small, and the thumb's movement distance in the X and Y planes is the largest. The judgment standard can be set to the thumb's movement distance being more than 5 times the index finger's movement distance. At the same time, combined with the thumb's movement direction on the X axis, it can be determined whether the user is flipping the phone forward or backward. Identification of key parts and motion recognition can be as follows Figure 4D shown.

[0101] Optionally, when the gesture operation is recognized, the device control method in the embodiment of the present application may further include:

[0102] The position information of the gesture operation is obtained, and the position information is transmitted to the terminal device, and the terminal device performs the corresponding operation according to the operation instruction and the position information. For example, after completing the recognition of the user's gesture posture and the calculation of the motion trajectory, in some special scenarios (such as opening an APP, answering a call, etc.), it is also necessary to combine the terminal device screen area to give the coordinate position of the user's operation.

[0103] In this way, with the help of the position information of the gesture operation, the object targeted by the user operation can be accurately determined, so that the corresponding operation can be accurately performed.

[0104] like Figure 5The specific implementation is as follows: the lower left corner of the mobile phone area where the screen is projected in the user's field of view is the coordinate origin, and the X and Y axis plane coordinate system is established; the upper right corner screen boundary coordinate point is defined to be consistent with the actual mobile phone screen resolution; after obtaining the relative position coordinates at the start and end of the user's hand movement, the operation instruction and the specific position information are transmitted to the mobile phone together, and the mobile phone performs subsequent processing. For example, the mobile phone can realize the functions or operations corresponding to the user's gestures in the VR environment according to the obtained user single-click, sliding and other operation instructions, combined with the synchronized position information, such as sliding the mobile phone screen, enlarging / reducing the size of the mobile phone display area, interface switching and other functions, so as to achieve the effect of remotely controlling the mobile phone end in the air. In addition, the above-mentioned mobile phone area can also be used as a designated area for transmitting pictures and videos from a VR device to a mobile phone.

[0105] In an embodiment of the present application, in order to ensure that the user's gesture operations can be accurately recognized without interference from other user actions, the distance and area of ​​the user's operation relative to the terminal screen video stream display area can be defined, and by judging the area and distance where the user's hand is located, the user's hand movements will only be recognized as valid instructions when they are in the set operation recognition area, thereby effectively avoiding misrecognition of user gestures and achieving remote and precise control of the terminal device.

[0106] Optionally, before the gesture operation is recognized, the device control method in the embodiment of the present application may further include:

[0107] Determine whether the gesture operation is located in a set operation recognition area; the operation recognition area can be pre-set based on actual conditions.

[0108] The above-mentioned recognition of the gesture operation includes: when the gesture operation is located in a set operation recognition area, the gesture operation is recognized. If the gesture operation is not located in the set operation recognition area, the gesture operation can be determined to be an invalid operation and does not need to be recognized.

[0109] In this way, with the help of the setting of the operation recognition area, misrecognition of user gestures can be effectively avoided.

[0110] Optionally, if the user's input operation is a voice input operation, the above step 13 may include:

[0111] The voice input operation is identified to obtain at least one of voice information, text information and operation instructions. Among them, the voice recognition and processing of the VR device may include at least three aspects: (1) The VR device receives voice data from the user and transmits the corresponding voice data to the terminal device after recognition; this is applicable to scenarios such as mobile phone calls, in which the mobile phone transmits the voice data received from the VR device to the call partner, thereby realizing mobile phone calls through the VR device; (2) Through the voice recognition service integrated in the VR device, the user's voice data is recognized in real time and converted into text information, and the text information is transmitted to the terminal device, and the terminal device performs corresponding operations, such as sending text messages, replying to messages, etc.; (3) Through the voice recognition service integrated in the VR device, the user's voice data is recognized in real time and converted into operation instructions, and the operation instructions are transmitted to the terminal device, and the terminal device performs corresponding operations, such as answering calls, opening client (APP), operating the screen, playing videos, pausing and dragging fast forward, etc.

[0112] Optionally, for the speech recognition of the VR device, a speech recognition technology may be used to convert speech into text, thereby recognizing the user's instructions. The speech recognition technology may be an existing speech recognition technology, or the existing speech recognition technology may be integrated into the VR device and implemented by calling a corresponding application programming interface API or toolkit SDK.

[0113] Optionally, for command parsing of VR devices, the user's instructions can be parsed by analyzing the recognized text. These instructions may include but are not limited to opening applications, making calls, sending text messages, searching for content, etc. According to actual needs, corresponding command parsing rules can be designed, for example, "open WeChat" is recognized as two instructions "open" and "WeChat", and subsequent processing is continued.

[0114] It should be noted that in order to improve the accuracy of voice control and user experience, voice data pre-processing and post-processing are required, such as noise reduction, dereverberation, feature extraction, etc. At the same time, voice recognition technology and command parsing rules need to be continuously optimized and improved to improve the performance and accuracy of the system.

[0115] See also Figure 6 , Figure 6 1 is a flow chart of a device control method provided in an embodiment of the present application. The method is applied to a terminal device, which may be, but is not limited to, a mobile phone, a watch, a tablet computer, etc. Figure 6 As shown, the method comprises the following steps:

[0116] Step 61: Transmit the collected screen video stream to the VR device, and the VR device displays the screen video stream.

[0117] In the embodiment of the present application, the terminal device can interact with the VR device through the APP therein.

[0118] The screen video stream may be obtained by recording, such as recording the real-time screen content and audio content of the terminal device. For example, after starting recording, the terminal device may use the MediaProjection API of the Android system or the ReplayKit of the iOS system to record the screen content in real time, and / or use the AudioRecord of the Android system or the AVAudioRecorder of the iOS system to record the audio in real time.

[0119] Optionally, in order to reduce the amount of data transmission, the terminal device (such as a mobile phone) can use a video encoder (such as H.264 or HEVC) to compress the collected video data and audio data, and use an audio encoder (such as Opus or AAC) to compress the audio data to obtain compressed audio and video data (i.e., screen video stream) and transmit it to the VR device.

[0120] In an optional implementation, the screen video stream can be transmitted from the terminal device to the VR device via a Socket connection between the terminal device and the VR device to ensure the stability and real-time performance of the transmitted data and to control the delay within an acceptable range, such as controlling the delay within tens of milliseconds.

[0121] Step 62: Obtain input information and / or operation instructions transmitted by the VR device; the input information and / or operation instructions are obtained according to the input operation performed by the user based on the screen video stream.

[0122] Optionally, the input operation may be a gesture operation or a voice input operation, which is not limited thereto. The input information may be voice information, text information, and the like.

[0123] In an optional embodiment, a terminal device (such as a mobile phone) can use monitoring to obtain voice information, text information, operation instructions, etc. from the VR device.

[0124] Step 63: Execute corresponding operations according to the input information and / or operation instructions.

[0125] Optionally, the corresponding operations performed by the terminal device may include but are not limited to making or receiving calls, opening an APP, operating the screen, playing a video, pausing a video, dragging and fast-forwarding a video, and the like.

[0126] Taking the mobile phone as an example, in the specific implementation, a Socket server can be created in the mobile phone to receive and parse the JSON format control commands / voices from the VR device, and trigger corresponding local operations according to the command type, such as making and receiving calls, sending text messages, opening APP, operating the screen, playing videos, pausing videos, and dragging to fast forward.

[0127] Through the scheme of the embodiment of the present application, the terminal device can transmit the collected screen video stream to the VR device, and the VR device can display the screen video stream; obtain the input information and / or operation instructions transmitted by the VR device; the input information and / or operation instructions are obtained according to the input operation performed by the user based on the screen video stream; and perform corresponding operations according to the input information and / or operation instructions. As a result, the terminal device (such as a mobile phone) can be controlled by the VR device, so that the user can also control other terminal devices while using the VR device, without removing the VR device or canceling VR-related functions, to achieve the effect of operating and using other terminal devices in the VR environment, thereby avoiding interruption of the user's VR experience, maximizing the continuity and consistency of the VR experience, and improving the user's VR experience.

[0128] In an embodiment of the present application, the terminal device (such as a mobile phone) mainly realizes the link with the VR device, establishes a two-way transmission channel between it and the VR device, and synchronizes the screen content and audio to the VR device via a wireless network. At the same time, it can also receive pictures, videos, etc. from the VR device, and receive instructions from the VR device, perform corresponding operations according to the instructions, thereby achieving the effect of remote control of the terminal device and realizing the integrated operation of the terminal device and the VR device.

[0129] The terminal device (such as a mobile phone) can perform corresponding operations according to the parsed instructions. For example, the API provided by the Android or iOS platform can be used to implement operations such as opening an application and making a call. If the user instruction is identified as: "Open" "APP2", the system interface can be called to open the application APP2. If the user instruction is identified as "Send a text message" "to" "Zhang San", the text message application can be opened and the recipient can be automatically filled in as Zhang San.

[0130] Optionally, before transmitting the captured screen video stream to the VR device, the device control method in the embodiment of the present application may further include:

[0131] A link request is sent to the VR device, and according to the link request, a bidirectional transmission channel is established between the VR device and the terminal device. According to this bidirectional transmission channel, the VR device can transmit instructions / information to the terminal device, and the terminal device can also transmit instructions / information to the VR device. The above link request can be triggered based on user operation. For example, after the user selects the VR device to be connected, the terminal device sends a link request to the selected VR device to establish a bidirectional transmission channel between the VR device and the terminal device.

[0132] The above-mentioned transmitting the collected screen video stream to the VR device includes: transmitting the screen video stream to the VR device through the bidirectional transmission channel. In this way, with the help of the bidirectional transmission channel, stable bidirectional communication between the VR device and the terminal device can be ensured.

[0133] Optionally, the device control method in the embodiment of the present application may further include:

[0134] Acquire the display content and control instructions of the VR device transmitted by the VR device, and process the display content according to the control instructions; wherein the control instructions are generated according to the control operation input by the user based on the display content.

[0135] In this way, operations that normally require the cooperation of the VR device and the terminal device can be changed to operations that only need to be performed on the VR side, without the need to operate multiple devices at the same time, thereby achieving the effect of controlling the terminal device through the VR device to process the images, audio and video displayed by the VR device itself, realizing the capability integration and joint control of the VR device and the terminal device, and improving user experience and operating efficiency.

[0136] Optionally, for the scenario of synchronizing the audio and video of the VR device to the terminal device (such as a mobile phone) for subsequent processing, according to the specific processing type, it can be divided into two categories: asynchronous processing and real-time synchronous processing; among them, asynchronous processing is suitable for scenarios that do not require real-time reading operations, such as the more common user scanning code login, scanning code payment and other scenarios; but in actual applications, there are scenarios that require real-time synchronous dynamic reading of videos or images, and the terminal device needs to adopt real-time synchronous processing. The two processing methods can be specifically described in the above embodiments, and will not be repeated here.

[0137] For example, taking mobile phones as an example, the mobile phone can use the two-way transmission channel between it and the VR device to obtain images, videos, real-time video streams, etc. from the VR device, and perform corresponding processing according to different processing strategies, such as saving them locally, outputting them to applications, and simulating the output of real-time images through virtual cameras, etc., thereby realizing the function of transmitting images and videos from VR to mobile phones.

[0138] The following takes the mobile phone as an example. Figure 7 The overall solution in this application is described in detail. Figure 7 As shown, the overall process may include:

[0139] Part 1, mobile phone to VR device: the mobile phone collects real-time screen content and audio content through its built-in APP, processes (such as compression) to obtain a screen video stream, and synchronizes the screen video stream to the VR device; the VR device receives the synchronized screen video stream and displays the mobile phone screen and sound in the user's current VR field of view; when the user performs hand movements on the mobile phone in the VR field of view, the VR device recognizes the user's gestures through its internal camera, generates control instructions based on the recognition results, and transmits the control instructions to the mobile phone APP; the mobile phone APP controls the mobile phone according to the received control instructions, such as hanging up the phone, sliding the screen, clicking the screen, etc. In this way, the screen interface of the mobile phone in the real world can be integrated into the virtual world, and the mobile phone can be manipulated or controlled by the VR device in combination with the user's gesture operation.

[0140] The second part, VR device to mobile phone: VR device can synchronize the content (such as video, audio, image, etc.) displayed by VR to the mobile phone APP by using the transmission channel between it and the mobile phone; the mobile phone APP locally saves the received content; when the user performs hand movements on the mobile phone in the VR field of view, the VR device recognizes the user's gestures through the camera inside it, and generates control instructions based on the recognition results, and transmits the control instructions to the mobile phone APP; the mobile phone APP processes the content transmitted by VR according to the received control instructions, such as scanning code to log in, pay, etc. In this way, for mobile phone applications that need to read external images in reverse, real-time video transmission and reading from VR to mobile phones can be realized; when users are doing VR-related activities, they can interact with the mobile phone in the current VR field of view without taking off the VR device, realize the joint control of the mobile phone and VR device, and completely integrate the relevant operations of the physical mobile phone with the operations of VR, so as to achieve a thorough integration of reality and virtuality.

[0141] In summary, the functions of VR devices can be expanded, and the content on the mobile phone screen can be displayed in the VR field of view, realizing the integration and joint operation of VR and mobile phone capabilities, avoiding frequent switching of terminal devices, and effectively improving the user's operating efficiency and actual experience.

[0142] It should be noted that the device control method provided in the embodiment of the present application can be executed by a device control apparatus or a control module in the device control apparatus for executing the device control method. In the embodiment of the present application, the device control apparatus provided in the embodiment of the present application is described by taking the device control apparatus executing the device control method as an example.

[0143] See also Figure 8 , Figure 8 is a schematic diagram of a device control device provided in an embodiment of the present application, which is applied to VR devices, such as Figure 8 As shown, the equipment control device 80 includes:

[0144] A first acquisition module 81 is used to acquire a screen video stream transmitted by a terminal device;

[0145] A display module 82, used for displaying the screen video stream through a VR device;

[0146] A second acquisition module 83, used for acquiring input information and / or operation instructions according to the input operation performed by the user based on the screen video stream;

[0147] The first transmission module 84 is used to transmit the input information and / or operation instruction to the terminal device, and the terminal device performs corresponding operations according to the input information and / or operation instruction.

[0148] Optionally, the second acquisition module 83 includes:

[0149] an identification unit, configured to identify the gesture operation when the input operation is a gesture operation, and obtain the gesture posture of the user and the movement trajectory information of the key parts of the user's hand under the gesture posture;

[0150] A generating unit is used to generate the operation instruction according to the gesture posture and motion trajectory information.

[0151] Optionally, the recognition unit is further used to: obtain position information of the gesture operation when recognizing the gesture operation;

[0152] The first transmission module 84 is further used to: transmit the location information to the terminal device, and the terminal device performs a corresponding operation according to the operation instruction and the location information.

[0153] Optionally, the equipment control device 80 further includes:

[0154] A judging module, used for judging whether the gesture operation is located in a set operation recognition area before recognizing the gesture operation;

[0155] The recognition unit is specifically configured to: when the gesture operation is located in the operation recognition area, recognize the gesture operation.

[0156] Optionally, the second acquisition module 83 is further used to: when the input operation is a voice input operation, identify the voice input operation to obtain at least one of voice information, text information and operation instructions.

[0157] Optionally, the equipment control device 80 further includes:

[0158] A first establishing module is used to establish a bidirectional transmission channel between the VR device and the terminal device in response to a link request of the terminal device before acquiring a screen video stream transmitted by the terminal device;

[0159] The first acquisition module 81 is specifically used to acquire the screen video stream through the bidirectional transmission channel.

[0160] Optionally, the equipment control device 80 further includes:

[0161] A second transmission module, configured to transmit the display content of the VR device to the terminal device in response to an input operation of the user;

[0162] A generating module, used for generating a control instruction according to a control operation input by a user based on displayed content;

[0163] The third transmission module is used to transmit the control instruction to the terminal device, and the terminal device processes the display content according to the control instruction.

[0164] The device control device 80 of the embodiment of the present application can realize the above Figure 1 The various processes of the method embodiment shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0165] See also Fig. 9 , Fig. 9 is a schematic diagram of the structure of a device control device provided in an embodiment of the present application, and the device is applied to a terminal device, such as a mobile phone, a watch, a tablet computer, etc.; Fig. 9 As shown, the equipment control device 90 includes:

[0166] A fourth transmission module 91 is used to transmit the collected screen video stream to a VR device, and the VR device displays the screen video stream;

[0167] A third acquisition module 92 is used to acquire input information and / or operation instructions transmitted by the VR device; wherein the input information and / or operation instructions are obtained according to the input operation performed by the user based on the screen video stream;

[0168] The execution module 93 is used to execute corresponding operations according to the input information and / or operation instructions.

[0169] Optionally, the equipment control device 90 further includes:

[0170] A sending module, used to send a link request to the VR device;

[0171] A second establishing module, configured to establish a bidirectional transmission channel between the VR device and the terminal device according to the link request;

[0172] The fourth transmission module 91 is specifically used to transmit the screen video stream to the VR device through the bidirectional transmission channel.

[0173] Optionally, the equipment control device 90 further includes:

[0174] A fourth acquisition module, used to acquire the display content and control instructions of the VR device transmitted by the VR device; wherein the control instructions are generated according to the control operation input by the user based on the display content;

[0175] A processing module is used to process the display content according to the control instruction.

[0176] The device control device 90 of the embodiment of the present application can realize the above Figure 6 The various processes of the method embodiment shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0177] Optional, such as Fig.10 As shown, the embodiment of the present application further provides an electronic device 100, including a processor 101, a memory 102, and a program or instruction stored in the memory 102 and executable on the processor 101. For example, when the electronic device 100 is a VR device, the program or instruction is executed by the processor 101 to implement the above Figure 1 When the electronic device 100 is a terminal device (such as a mobile phone), the program or instruction is executed by the processor 101 to implement the above Figure 6 The various processes of the device control method embodiment shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0178] The present application also provides a computer program product, including computer instructions, which can achieve the above-mentioned Figure 1 or Figure 6 The various processes of the method embodiment shown can achieve the same technical effect, and to avoid repetition, they will not be described again here.

[0179] An embodiment of the present application also provides a readable storage medium on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned device control method embodiment can be implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0180] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0181] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0182] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0183] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a service classification device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0184] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A device control method, characterized in that: include: Get the screen video stream transmitted by the terminal device; Displaying the screen video stream through a virtual reality VR device; Acquiring input information and / or operation instructions according to an input operation performed by a user based on the screen video stream; The input information and / or operation instruction is transmitted to the terminal device, and the terminal device performs corresponding operations according to the input information and / or operation instruction.

2. The method according to claim 1, characterized in that If the input operation is a gesture operation, the acquiring of input information and / or operation instructions according to the input operation performed by the user based on the screen video stream includes: Recognize the gesture operation to obtain the user's gesture posture and movement trajectory information of key parts of the user's hand under the gesture posture; The operation instruction is generated according to the gesture posture and the motion trajectory information.

3. The method according to claim 2, characterized in that When the gesture operation is recognized, the method further includes: Obtaining position information of the gesture operation; The location information is transmitted to the terminal device, and the terminal device performs corresponding operations according to the operation instruction and the location information.

4. The method according to claim 2, characterized in that Before identifying the gesture operation, the method further includes: Determine whether the gesture operation is located in a set operation recognition area; Wherein, the identifying the gesture operation includes: When the gesture operation is located in the operation recognition area, the gesture operation is recognized.

5. The method according to claim 1, characterized in that: Before obtaining the screen video stream transmitted by the terminal device, the method further includes: In response to a link request from the terminal device, establishing a bidirectional transmission channel between the VR device and the terminal device; The step of obtaining the screen video stream transmitted by the terminal device includes: The screen video stream is obtained through the bidirectional transmission channel.

6. The method according to claim 1, characterized in that The method further comprises: In response to an input operation by a user, transmitting the display content of the VR device to the terminal device; generating a control instruction according to a control operation input by a user based on the displayed content; The control instruction is transmitted to the terminal device, and the terminal device processes the display content according to the control instruction.

7. A device control method, characterized in that: include: Transmitting the collected screen video stream to a VR device, and having the VR device display the screen video stream; Acquire input information and / or operation instructions transmitted by the VR device; wherein the input information and / or operation instructions are obtained according to input operations performed by the user based on the screen video stream; Execute corresponding operations according to the input information and / or operation instructions.

8. The method according to claim 7, characterized in that Before transmitting the collected screen video stream to the VR device, the method further includes: Sending a link request to the VR device, and establishing a bidirectional transmission channel between the VR device and the terminal device according to the link request; The step of transmitting the collected screen video stream to the VR device includes: The screen video stream is transmitted to the VR device through the bidirectional transmission channel.

9. An electronic device, characterized in that: The method comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method according to any one of claims 1 to 6, or implements the steps of the method according to claim 7 or 8.

10. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 6, or implement the steps of the method according to claim 7 or 8.

Images