Interaction control method, device, equipment, medium and program product

Abstract

The embodiment of the invention provides an interaction control method, an interaction control device, interaction control equipment, a medium and a program product, the method is executed by main control equipment, wireless connection is established between the main control equipment and controlled equipment, and the method comprises the following steps: displaying at least two window interfaces in an interaction control scene, each window interface corresponds to one screen of the controlled equipment and presents screen content; monitoring an input event generated when an indication cursor of the input device moves from the first window interface to the second window interface, generating input event information of the input event, and sending the input event information to the controlled device, so that the controlled device determines that a dragging operation is performed on the target object based on the input event information; and receiving and displaying picture content for dragging the target object from the first screen to the second screen. By means of the method, the main control device can effectively control the cross-screen dragging movement of the target object on the controlled device, and the input control flexibility and simplicity of the main control device on the controlled device are better reflected.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer application technology, and in particular, to an interactive control method, device, equipment, medium and program product. Background Art

[0002] At present, a master control device can be used to interactively control a controlled device. The master control device can be a virtual reality device that is different from traditional electronic devices (such as mobile terminals and computers). Compared with the interactive control of the controlled device by the traditional master control device, the master control device provides more abundant input methods, such as gesture input and handle input, to simulate the mouse behavior for remote operation.

[0003] With the continuous development of virtual reality technology, it has the function of multi-window display. If the controlled device is configured with two or more screens, the interface content displayed on each screen can be presented separately in the multiple window interfaces of the main control device.

[0004] However, in the existing interactive control implementation, the main control device cannot flexibly realize the cross-screen presentation of the controlled object on the controlled device from one screen to another, which affects the operator's operating experience. Summary of the invention

[0005] The present disclosure provides an interactive control method, apparatus, device, medium and program product to achieve flexibility and simplicity in input control of a master control device over a controlled device.

[0006] In a first aspect, an embodiment of the present disclosure provides an interactive control method, which is applied to a master control device, wherein the master control device establishes a wireless connection with a controlled device, and the controlled device includes at least two screens, and the method includes:

[0007] In the interactive control scenario, at least two window interfaces are displayed, each window interface corresponds to a screen of the controlled device, and the screen content of the corresponding screen is presented;

[0008] monitoring an input event generated by an input device, generating input event information of the input event, and sending the input event information to the controlled device, so that the controlled device determines that a drag operation has occurred on the target object based on the input event information, wherein the input event is generated when an indication cursor of the input device moves from the first window interface to the second window interface;

[0009] The screen content fed back by the controlled device is received and displayed, wherein the screen content is a target object dragged from a first screen to a second screen, and the first screen and the second screen correspond to the first window interface and the second window interface respectively.

[0010] In a second aspect, an embodiment of the present disclosure provides an interactive control method, which is applied to a controlled device, wherein the controlled device establishes a wireless connection with a master device, and the controlled device includes at least two screens, and the method includes:

[0011] In the interactive control scenario, the screen contents of each of the screens are sent to the main control device, so as to display at least two window interfaces on the main control device, each of which corresponds to a screen and presents the screen contents of the corresponding screen;

[0012] receiving input event information sent by the master control device, and determining, according to the input event information, that a drag operation has occurred on the target object, wherein the input event information is generated by the master control device by monitoring input events of an input device;

[0013] Respond to the drag operation, and feed back the screen content generated in the drag operation response to the main control device, so that the main control device receives and displays the screen content, wherein the screen content is the target object dragged from the first screen to the second screen, and the first screen and the second screen correspond to the first window interface and the second window interface on the main control device respectively.

[0014] In a third aspect, an embodiment of the present disclosure further provides an interactive control device, which is configured in a master control device, wherein the master control device establishes a wireless connection with a controlled device, wherein the controlled device includes at least two screens, and the device includes:

[0015] A window display module, used to display at least two window interfaces in the interactive control scenario, each window interface corresponding to a screen of the controlled device, and present the screen content of the corresponding screen;

[0016] an information sending module, configured to monitor an input event generated by an input device, generate input event information of the input event, and send the input event information to the controlled device, so that the controlled device determines that a drag operation has occurred on the target object based on the input event information, wherein the input event is generated in the process of an indication cursor of the input device moving from the first window interface to the second window interface;

[0017] The content display module is used to receive and display the screen content fed back by the controlled device, wherein the screen content is a target object dragged from a first screen to a second screen, and the first screen and the second screen correspond to the first window interface and the second window interface respectively.

[0018] In a fourth aspect, an embodiment of the present disclosure provides an interactive control device, which is configured on a controlled device, wherein the controlled device establishes a wireless connection with a main control device, and the controlled device includes at least two screens, and the device includes:

[0019] A content sending module, used for sending the screen content in each of the screens to the main control device in the interactive control scenario, so as to display at least two window interfaces on the main control device, each window interface corresponding to a screen, and presenting the screen content in the corresponding screen;

[0020] An information receiving module, used to receive input event information sent by the master control device, and determine whether a drag operation has occurred according to the input event information, wherein the input event information is generated by the master control device by monitoring input events of the input device;

[0021] An operation response module is used to respond to the drag operation and feed back the screen content generated in the drag operation response to the main control device so that the main control device receives and displays the screen content, wherein the screen content is the target object dragged from the first screen to the second screen, and the first screen and the second screen correspond to the first window interface and the second window interface on the main control device respectively.

[0022] In a fifth aspect, an embodiment of the present disclosure provides a virtual reality device, including:

[0023] one or more input devices;

[0024] one or more processors;

[0025] a storage device for storing one or more programs,

[0026] The one or more programs are executed by the one or more processors, so that the one or more processors implement the interactive control method provided in the first aspect of this embodiment.

[0027] In a sixth aspect, an embodiment of the present disclosure provides an electronic device, including:

[0028] one or more processors;

[0029] a storage device for storing one or more programs,

[0030] The one or more programs are executed by the one or more processors, so that the one or more processors implement the interactive control method provided in the second aspect of this embodiment.

[0031] In a seventh aspect, an embodiment of the present disclosure further provides a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute the interactive control method described in any embodiment of the present disclosure.

[0032] In an eighth aspect, an embodiment of the present disclosure further provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the interactive control method described in any embodiment of the present disclosure is implemented.

[0033] An interactive control method, apparatus, device, medium and program product provided by the embodiments of the present disclosure, the method involves interactive control between a master control device and a controlled device, wherein a wireless connection is established between the master control device and the controlled device and the controlled device includes at least two screens, wherein the master control device first displays at least two window interfaces in the interactive control scenario in which it is located, each window interface corresponds to a screen of the controlled device, and presents the screen content in the corresponding screen; then, the master control device monitors input events generated by an input device, generates input event information of the input events, and sends the input event information to the controlled device, so that the controlled device determines that a drag operation has occurred on a target object based on the input event information, wherein the input event is generated when an indicating cursor of the input device moves from a first window interface to a second window interface; and finally, the screen content fed back by the controlled device is received and displayed, wherein the screen content is the target object being dragged from the first screen to the second screen, and the first screen and the second screen correspond to the first window interface and the second window interface, respectively. In the above technical solution of this embodiment, the master device acts as the input event generating end of the controlled device, and can provide the controlled device with operation events for interactive control, so that the controlled device can parse the operation logic corresponding to the operation event and execute it. The execution result of the operation logic can be displayed on the master device, so that the master device displays the same screen content as the controlled device. Compared with the prior art, the above technical solution of this embodiment can realize the continuous reception of input events generated by the input device by the master device, and will not interrupt the reception of input events due to the input device moving from one window to another. The continuous reception of input events generated across windows ensures that the controlled device can also continuously respond to the operation instructions corresponding to the input events, thereby ensuring the cross-screen dragging control of the controlled object on the controlled device by the master device, simplifying the existing cross-screen dragging that requires multiple interactions, and also improving the input control flexibility of the master device, and also improving the operator's operating experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0035] Figure 1a and Figure 1b A diagram showing the effect of dragging a controlled object across the screen in existing interactive control implementations is given;

[0036] Figure 2a A flow chart of an interactive control method provided by an embodiment of the present disclosure is given;

[0037] Figure 2b A diagram showing the effect of dragging across the screen according to an interactive control method provided by an embodiment of the present disclosure is provided;

[0038] Figure 3 A flow chart of an interactive control method provided by an embodiment of the present disclosure is given;

[0039] Figure 4 A structural schematic diagram of an interactive control device provided by an embodiment of the present disclosure is given;

[0040] Figure 5 A structural schematic diagram of an interactive control device provided by an embodiment of the present disclosure is given;

[0041] Figure 6 A structural schematic diagram of a virtual reality device provided by an embodiment of the present disclosure is given;

[0042] Figure 7 A schematic structural diagram of an electronic device provided by an embodiment of the present disclosure is given. DETAILED DESCRIPTION

[0043] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0044] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0045] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0046] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0047] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0048] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0049] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0050] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0051] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0052] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0053] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0054] It should be noted that the application scenario involved in this embodiment can be described as: in the scenario where a wireless connection is established between the virtual reality device and the computer, the virtual reality device can be used as a master device to perform input control on the computer as a controlled device. This is equivalent to inputting control instructions for the screen content of the computer through the virtual reality device. When the virtual reality device has a multi-window display function, if the computer has multiple screens, the virtual reality device can use the multi-window function to display the screen content of each screen through a window interface.

[0055] Assuming that the screen content of each screen on the computer side has been displayed in each window interface on the virtual reality device side, if the operator controls the screen content on the computer side through the virtual reality device, and specifically wants to realize the interactive control of dragging the controlled object on one screen of the computer side to another screen across the screen, through the existing interactive control method, the operator needs to perform at least two operations on the controlled object. Among them, one is to drag the controlled object from its initial position on the first screen to the edge of the first screen. If all the contents of the controlled object are still in the first screen after dragging, it is necessary to drag the controlled object again in the first screen to verify that part of the contents of the controlled object falls into the second screen. Then, it is necessary to at least drag the controlled object that falls into the second screen in the second screen to ensure that all the contents of the controlled object fall into the second screen.

[0056] For example, Figure 1a and Figure 1b The following diagram shows the effect of dragging the controlled object across the screen in the existing interactive control implementation. Figure 1a and Figure 1b The description is presented from the perspective of virtual reality devices. Figure 1a and Figure 1b The virtual display screen 1 of the virtual reality device is shown in the figure. The virtual display screen 1 shows two window interfaces, namely the first window interface 11 and the second window interface 12, and also includes a controlled object 13 as an interactive control object. According to the existing interactive control implementation method, when the controlled object 13 is dragged from the first window interface 11 (corresponding to the first screen of the computer end) to the second window interface 12 (corresponding to the first screen of the computer end) through the input form of the gesture 14, after the controlled object 13 is pressed and moved at least once, the following will be presented first. Figure 1a The effect shown is that only a part of the area of ​​the controlled object 13 falls into the second window interface 12 .

[0057] If the gesture 14 is still in the state of pressing and moving without pressing and lifting, the virtual reality device will no longer recognize the pressing and moving recognition due to the limitation of the existing interactive control implementation. Therefore, no matter how the gesture 14 is moved in the second window interface 12, the controlled object 13 will still stay in the state of pressing and moving. Figure 1a The state shown is that a part of the area of ​​the controlled object remains in the first window interface 11, and the other part of the area remains in the second window interface 12 and does not continue to move to the second window interface.

[0058] If you want to completely drag the controlled object to the second window interface, you need to first generate a press-and-lift event through gesture 14, then apply gesture 14 to a partial area of ​​the controlled object in the second window interface 12, and generate a press-and-move event again. After gesture 14 stops moving and lifts up, the press-and-move event generated again in the second window interface can be received and recognized by the virtual reality device. Only by receiving the press-and-move event again can the dragging of the controlled object 13 be responded to again, so that the controlled object area remaining in the first window interface can be completely dragged to the second window interface 12, presenting Figure 1b The effect shown.

[0059] It can be seen from this that the existing interactive control cannot flexibly realize the cross-screen dragging of the controlled object, and its implementation process is relatively cumbersome, which affects the operator's operating experience.

[0060] Based on this, Figure 2a This is a flow chart of an interactive control method provided in an embodiment of the present disclosure. This embodiment can be applicable to situations where interactive control is performed with a controlled device. The method can be executed by an interactive control device, which can be implemented by software and / or hardware and can be configured in an electronic device serving as a master control device. Specifically, the electronic device can preferably be a virtual reality device.

[0061] The master device can establish a wireless connection with the controlled device, and the controlled device includes at least two screens, such as Figure 2a As shown, an interactive control method provided in this embodiment may include the following steps:

[0062] S201. In an interactive control scenario, display at least two window interfaces, each window interface corresponding to a screen of the controlled device, and presenting screen content in the corresponding screen.

[0063] In this embodiment, it can be considered that after the master device and the controlled device establish a wireless connection, the interactive control of the master device over the controlled device is triggered, and thus it is considered that the master device has entered the interactive control scene. At the same time, the interactive control involved in this embodiment can be considered as the implementation description of the interactive control after the master device is in the interactive control scene.

[0064] This step can be considered as the relevant presentation of the main control device when it is in an interactive control scenario and no input control is performed. Specifically, the screen content of the controlled device can be displayed on the display device of the main control device. It can be known that in this embodiment, it is preferred that the main control device has a multi-window display function. When the controlled device has at least two screens, the main control device can display at least two window interfaces through this step to respectively present the screen content of each screen on the controlled device.

[0065] It should be noted that when the main control device is a virtual reality device, the display device it has can be considered as a virtual display screen, and each window interface can be displayed on the virtual display screen.

[0066] S202. Listen to input events generated by an input device, generate input event information of the input event, and send the input event information to the controlled device so that the controlled device determines that a drag operation has occurred on a target object based on the input event information, wherein the input event is generated when an indicating cursor of the input device moves from a first window interface to a second window interface.

[0067] In this embodiment, the input device can be considered as a device connected to the active device for performing information operation events, which can be a keyboard, mouse, touchpad, etc., or a handle and gestures, etc.

[0068] It should be noted that the operating system on which this main control device relies has a built-in general input receiving function for monitoring input events generated by input devices. However, this general input receiving function has a limitation, that is, this general input receiving function can only ensure the normal reception of input events generated by input devices when this main control device is in a normal application scenario. When this main control device is in an interactive control scenario, the various window interfaces it displays represent multiple screens of the controlled device. In this case, the general input receiving function originally configured by this main control device is only allowed to receive press-and-move events generated in any window interface. If the input device moves out of the window interface during the process of generating the press-and-move event, the general input receiving function will terminate the reception of subsequent press-and-move events generated by the input device.

[0069] Based on the functional limitations of the general input receiving function, it can be known that the general input receiving function cannot complete the reception of all the press-and-move events generated in the cross-window interface scenario. This is equivalent to the fact that the main control device cannot monitor the complete press-and-move event in the cross-window interface press-and-move scenario, resulting in the controlled terminal being unable to respond to the completed press-and-move operation based on the completed press-and-move event.

[0070] Therefore, it can be known from the above description that in order for the controlled device to realize the cross-screen drag operation according to the press-move event of the main control device, it is necessary to ensure that the main control device monitors the complete press-move event in the scenario of pressing and moving across the window interface. This embodiment can realize the complete monitoring of the input event generated by the input device in the interactive control scenario through this step. Even if the input device indicates that the cursor moves from the first window interface to the second window interface during the process of generating the input event, the input event generated by the input device in the first window interface and the input event generated in the second window interface can be monitored respectively through this step.

[0071] Specifically, one way to implement the monitoring of input events of the input device in this step can be described as: by constructing an input receiving function additionally relative to the interactive control scenario in the dependent operating system, continuous monitoring of input events generated by the input device can be implemented. The input receiving function used in this embodiment can be considered to be specially constructed relative to the interactive control scenario, which redefines the reception of each input event in the input receiving function, and does not limit the receivable area to each window interface.

[0072] In this embodiment, it is also necessary to generate corresponding input event information for the monitored input event through this step. The input event information can be considered as information describing the event attribute characteristics of the input event. The input event information can be used to respond to the input event at the software level, and is also the information that the controlled device relies on to respond to the input event.

[0073] It should be noted that the input device is often an external hardware device of the main control device, and the input events transmitted to the main control device are often represented in a format compatible with the input device. The event information represented by each input event cannot be directly recognized by the operating system that the main control device relies on.

[0074] Therefore, an implementation method of this step to generate input event information relative to an input event can be described as: first obtain the event attribute information carried by the input device when generating an input event, and then use the pre-set event mapping information for converting the data format involved in the hardware data with the software-readable format involved in the operating system to determine the event description information corresponding to the event attribute information at the software level, and the event description information can be determined as the input event information of the input event.

[0075] In addition, in this step, the generated input event information may be sent to the controlled device, so that the main control device can implement input control of the controlled device through the sent input event information.

[0076] Furthermore, in this embodiment, the main control device may be preferably a virtual reality device; the input device package may be preferably a handle, and the indicating cursor may be a handle ray; or, the input device may be a gesture, and the indicating cursor may be a hand icon; or, the input device may be a Bluetooth mouse, and the indicating cursor may be a mouse arrow.

[0077] Among them, virtual reality devices are usually presented in a head-mounted form or a glasses form in actual applications. The paired input device can be a handle. After the handle is connected to the virtual reality device, a ray-style indicator cursor of the handle can be presented on the virtual display screen of the virtual reality device. The paired input device can also include gestures, that is, using a camera or infrared device to sense the player's hands, establish an input mapping with the hands, and present the gesture icon corresponding to the hand on the virtual display screen as an indicator cursor; at the same time, the paired input device can also be a Bluetooth mouse, and the corresponding Bluetooth mouse can present an indicator cursor of a mouse arrow.

[0078] It is known that when the input device generates a move or press move input event, the pointer of the input device will change position along with the movement. Therefore, the pointer of the input device may move from the first window interface to the second window interface.

[0079] S203: Receive and display screen content fed back by the controlled device, where the screen content is a target object dragged from the first screen to the second screen, and the first screen and the second screen correspond to the first window interface and the second window interface, respectively.

[0080] It should be noted that the control of the controlled device by the main control device is mainly achieved by generating input events and generating input event information and sending it to the controlled device. In addition, the main control device can also serve as a synchronous display end of the controlled device. While the controlled device executes the logical operations involved in the input events, it can also synchronously feedback the screen information corresponding to the logical operations to the main control device, so that the screen content on the controlled device can be synchronized to the main control device in real time for display.

[0081] In this embodiment, the input event information sent to the controlled device can be recognized by the controlled device, thereby determining that an actual drag operation has occurred on the target object based on the input event information sent by the main control device, and specifically determining that a drag operation has been performed on the target object from the first screen to the second screen; the controlled device can then respond to the drag operation and present the screen content of the target object being dragged from the first screen to the second screen.

[0082] In order to ensure the real-time presentation of the screen content of the controlled device on the main control device, this step can be used to receive the screen content generated by the controlled device in response to the drag operation in real time and display it in real time. The screen content presents the process of dragging the target object from the first screen to the second screen on the controlled device. From the above description, it can be seen that the first screen corresponds to the first window interface, and the second screen corresponds to the second window interface. Accordingly, the process of dragging the target object from the first window interface to the second window interface can be displayed on the main control device.

[0083] The above technical solution of this embodiment, because the input events generated during the process of the pointer cursor of the input device moving from the first window interface to the second window interface are all completely monitored through the above S202, therefore, the input event information generated relative to the input event and sent to the controlled device is also correspondingly complete, so the continuity of the target object moving from the first screen to the second screen is guaranteed, which is equivalent to flexibly realizing the cross-screen drag operation of the controlled object. Compared with the prior art, the above technical solution of this embodiment can realize the continuous reception of the input events generated by the input device by the main control device, and will not interrupt the reception of the input events due to the input device moving from one window to another window. The continuous reception of the input events generated across windows ensures that the controlled device can also continuously respond to the operation instructions corresponding to the input events, thereby ensuring the cross-screen drag control of the controlled object on the controlled device by the main control device, simplifying the existing cross-screen drag that requires multiple interactions to achieve, and also improving the input control flexibility of the main control device, and also improving the operator's operating experience.

[0084] As a first optional embodiment of this embodiment, based on the above embodiment, monitoring the input event generated by the input device and generating the input event information of the input event can be further optimized into the following steps:

[0085] a1) monitoring the pointing cursor of the input device by calling an input receiving function, wherein the input receiving function is constructed in the operating system that the main control device relies on, and is constructed in association with the input receiving function and the interactive control scene.

[0086] It can be known that this first optional embodiment provides an implementation of monitoring input events and an implementation of generating input event information. Among them, this first optional embodiment can be used to implement the monitoring logic of input events. Specifically, this step can be used to call the input receiving function pre-built in the operating system that the main control device relies on. The input receiving function can be considered to be independent of the general input function originally possessed by the operating system, and is an input function that is additionally constructed relative to the interactive control scenario.

[0087] Furthermore, the input receiving function is constructed in a function library of the operating system, and the input receiving function defines a code description for establishing an association with the interactive control scenario; after the wireless connection is established, the input receiving function is called and started running through a set input calling interface.

[0088] In this embodiment, the relevant code description of the input receiving function can be called and triggered to start running after the master device and the controlled device establish a wireless connection. Among them, the input receiving function can be called through the input calling interface constructed in the dependent operating system. In this embodiment, the operating system that the master device relies on can preferably be an Android system.

[0089] After calling the input receiving function that has been started, this step can monitor the pointer of the input device. By monitoring the position change of the pointer, it can be determined whether the input device has generated a move event, or whether a press and move event has occurred.

[0090] b1) receiving a first input event generated by the input device when the indicating cursor is in the first window interface.

[0091] In this embodiment, this step can analyze the initial position of the pointer, and then when the pointer moves or is pressed, it can be determined that the input device has generated an input event, thereby receiving the generated input event, and this embodiment records the input event as the first input event. The first input event can be a moving event or a moving event in a pressed state, and the specific event of the first input event can be determined based on the event attribute information it carries.

[0092] c1) When the indicating cursor moves from the first window interface to the second window interface, continue to receive a second input event generated by the input device.

[0093] In this embodiment, by monitoring the indicating cursor through the above steps, it can also be determined that the position of the indicating cursor has moved, and specifically moved out of the first window interface and then moved into the second window interface. In this case, the input event generated by the input device in the second window interface can continue to be received through the input receiving function. This embodiment can record the input event as a second input event.

[0094] The above technical solution of this embodiment provides a complete reception implementation of obtaining the output events generated by the input device when it moves from one window interface to another window interface. Different from the prior art that can only receive input events generated in one window interface, the above solution of this embodiment better ensures the integrity of the received input events and also ensures the simplification of operation events at the operation level. This embodiment provides basic data support for the subsequent realization of flexible dragging of controlled objects across screens.

[0095] As a second optional embodiment of this embodiment, based on the above embodiment, the input event information for generating the input event can be further optimized into the following steps:

[0096] a2) Acquire event attribute information corresponding to the input event, wherein the event attribute information includes: original input coordinates of the event and a pressed state.

[0097] It can be known that this second optional embodiment can be used to implement the generation logic of input event information involved in the input event. Specifically, this step can be used to obtain the event attribute information generated by the input device when the input event is generated. The event attribute information may include the pressing state of the input device when the input event is generated, and the original input coordinates of the event. The original input coordinates of the event can be considered as the coordinate information on the display screen corresponding to the main control device, and the coordinate system of the display screen is used for coordinate marking.

[0098] It should be noted that the event attribute information generated when the input device generates an input event may also include information such as the input device type, device quiet flag, and event generation timestamp. Through the included information, it is clear which input device generated the input event this time, and it is also possible to know more specifically which input key of the input device generated the input event, and know the specific time point at which the input event was generated. The above information can also be used to further determine the operation event to ensure the accuracy of the operation event determination.

[0099] b2) Determining input event information corresponding to the input event according to the preset first event mapping information and the event attribute information.

[0100] It can be known from the above description that the event attribute information obtained in the above steps can be considered as event information formed in a format recognizable by the input device. As an external device, the event attribute information generated by the input device cannot be directly recognized by the operating system that the main control device relies on. Based on this, this optional embodiment can convert the event attribute information through this step to obtain input event information recognizable by the operating system.

[0101] Among them, the first event mapping information can be considered as information for mapping the data format involved in the input device and the format readable by the operating system. In this embodiment, the event mapping information is preferably recorded as the first event mapping information, and the first event mapping information includes the mapping conversion between the input device and the operating system.

[0102] Specifically, in this optional embodiment, as one implementation method, determining the input event information corresponding to the input event according to the preset first event mapping information and the event attribute information can be optimized into the following steps:

[0103] b21) Determine the operation event represented by the input event according to the original input coordinates and the pressing state of the event.

[0104] In this embodiment, it can be known that each input event corresponds to an event original input coordinate and a pressed state. After monitoring a certain number of input events, it is possible to determine whether the input event includes a move operation based on whether the event original input coordinates of the monitored input event change, or it is possible to determine whether a press operation occurs based on the pressed state of the input event. This step can determine the corresponding operation event through the above-mentioned judgment of the input event. The operation event can be press, move, or press and move, etc.

[0105] b22) Generate event description information of the operation event according to the first event mapping information, and use the event description information as the input event information.

[0106] In this embodiment, after determining the operation event actually possessed by the input device through the generated input event through the above steps, event description information recognizable at the operating system software level can be generated for the operation event based on the first event mapping information. The event description information is equivalent to describing the event attributes of the input event at the software level, and also carries information describing the input event such as the original input coordinates of the event and the pressed state. This step can ultimately determine the formed event description information as input event information.

[0107] It should be noted that, based on the fact that the event attribute information includes the input device type, device button identifier, and event generation timestamp, the above information converted into event description information can be carried in the input event information formed. The above information is sent to the controlled device to ensure more accurate analysis of the input event.

[0108] The above technical solution of this embodiment provides an implementation for generating input event information. The generated input event information provides basic data support for subsequent input control of the controlled device.

[0109] In order to better understand the interactive effect brought by the interactive control method provided in this embodiment, Figure 2b The following is a diagram showing the effect of dragging across the screen according to an interactive control method provided by an embodiment of the present disclosure. Figure 2b As shown, it can also be considered that the display and description are from the perspective of the virtual reality device. Specifically, the virtual display screen 2 corresponding to the virtual reality device is shown. The virtual display screen 2 also displays two window interfaces, namely the first window interface 21 and the second window interface 22, and also includes a controlled object 23 as an input control object. It can be seen that the controlled object 23 is initially presented in the first window interface 21.

[0110] When interactive control is implemented through the interactive control method provided in this embodiment, input events generated by the input device can be continuously monitored. Therefore, in the implementation of dragging the controlled object 23 from the first window interface 21 to the second window interface 22 across windows, all input events (press and move events) generated by the input device's indicating cursor 24 (gesture icon) moving from the first window interface 21 to the second window interface 22 can be continuously received. Therefore, the input event information of all input events can also be determined and sent to the controlled device to respond to the input control of the main control device. The controlled device can parse each input event information to determine that a drag operation has occurred on the controlled object 23. At the same time, the virtual reality device can also synchronously receive the screen content generated by the controlled device in the process of responding to the drag operation, and display it in real time. Figure 2a The effect shown is equivalent to showing the interactive control effect of dragging the controlled object 23 into the second window interface through a drag operation.

[0111] Different from the interactive control in the prior art which requires multiple drags to drag the controlled object to another window interface, the interactive control method provided in this embodiment realizes the interactive control of the controlled object more simply and flexibly, and effectively improves the operator's input control operation experience.

[0112] Figure 3 A flow chart of an interactive control method provided in an embodiment of the present disclosure. The embodiment of the present disclosure is applicable to situations where interactive control is performed with a main control device. The method can be executed by an interactive control device, which can be implemented in the form of software and / or hardware. Optionally, it can be configured in an electronic device serving as a controlled device, which can be a mobile terminal, a PC, a server, etc.

[0113] The controlled device can establish a wireless connection with the main control device, and the controlled device as the execution subject of this embodiment can be configured with at least two screens. Figure 3 As shown, an interactive control method provided in this embodiment may include the following steps:

[0114] S301. In the interactive control scenario, the screen contents of each of the screens are sent to the main control device, so as to display at least two window interfaces on the main control device, each window interface corresponding to one screen, and presenting the screen contents of the corresponding screen.

[0115] In this embodiment, it can be considered that after the controlled device establishes a wireless connection with the master device, it is equivalent to authorizing the master device to control the input and synchronous display functions of the controlled device, and it can also be considered that the controlled device has entered the interactive control scene. Among them, after this embodiment is in the interactive control scene, it can respond to the interactive control of the master device on the controlled device, so it is considered that the interactive control logic involved in the controlled device is realized after it is in the interactive control scene.

[0116] This step can be considered as the relevant presentation of synchronizing the screen contents on all available screens to the master control device for display after being in the interactive control scene and before receiving new input control sent by the master control device. This step can specifically send the screen contents on each screen to the master control device respectively, and the master control device can receive the screen contents of each screen and display them respectively on the window interface corresponding to the screen.

[0117] S302: Receive input event information sent by the master control device, and determine, according to the input event information, that a drag operation has occurred on a target object, wherein the input event information is generated by the master control device by monitoring input events of an input device.

[0118] It can be known that in the existing interactive control implementation, since the master control device is limited to receiving the press-and-move event on one window interface, if the press-and-move event generated by the input device crosses windows, the master control device can only receive the input event generated in the first window interface, and after the pointer of the input device moves out of the window interface, the input event generated after the pointer falls into the second window interface is terminated. Therefore, it is equivalent to that the input event received by the master control device is incomplete during the process of the pointer of the input device sending the cross-window movement.

[0119] Considering that the controlled device responds to the interactive control logic based on the input event received by the master device, if the master device cannot receive the complete input event, the controlled device cannot execute the complete interactive control logic based on the received input event information. As a result, the problem that the controlled object cannot achieve cross-screen through a single drag occurs.

[0120] Through the interactive control method provided for the main control device in the present embodiment, it can be ensured that the main control device can continuously receive input events even when the input device generates input events when moving across windows, thereby receiving all input events generated by the input device during the process of moving across windows, thereby ensuring the integrity of the received input events.

[0121] At the same time, the master device can also generate input event information for each received input event and send it to the controlled device. The controlled device can receive the input event information of each input event through this step, and then determine that a drag operation has occurred on the target object by analyzing the input event information. It can be considered that the input event involved in the drag operation on the target object is monitored by the master device and the input event information is generated accordingly.

[0122] Specifically, an implementation for determining whether a drag operation has occurred on a target object can be described as follows: by analyzing the input event information, the input event screen coordinates corresponding to the input event on the screen of the controlled device and the operation event corresponding to the input event screen coordinates are determined; when the operation event is determined to be a press-to-move, the starting position, the ending position and the target object involved in the press-to-move can be determined; when the starting position is on the first screen, the target object is also on the first screen, and the ending position is on the second screen, it can be determined that a drag operation has occurred from the first screen to the second screen on the target object.

[0123] It should be noted that the input event information received by the controlled device is described in a description method that is recognizable by the master device or associated with the master device, but the input event information under this description method may not be directly recognizable by the controlled device. Therefore, an event mapping information can be used to convert the input event information to obtain device event description information that is recognizable by the controlled device.

[0124] S303, responding to the drag operation, and feeding back the screen content generated in the drag operation response to the main control device, so that the main control device receives and displays the screen content, wherein the screen content is the target object dragged from the first screen to the second screen, and the first screen and the second screen correspond to the first window interface and the second window interface on the main control device respectively.

[0125] It can be known that after the controlled device determines through the above steps that the interactive control involved in the input event information involves a drag operation, specifically involving a drag operation of the target object from the first screen to the second screen, it can respond to the drag operation through this step, so as to present the screen content of the target object being dragged from the first screen to the second screen in response to the drag operation, and the screen content can be synchronously fed back to the main control device to display the screen of the target object being dragged from the first window interface to the second window interface on the main control device. Among them, the first window interface corresponds to the first screen, and the second window interface corresponds to the second screen.

[0126] The interactive control method provided in this embodiment specifically provides a real-time response process to the press-and-move event when the controlled device side presses and moves across windows through the master device side. In the above embodiment, it is ensured that the master device will not interrupt the reception of input events due to the input device moving from one window to another window, and on the basis of being able to continuously receive input events generated across windows, it is ensured that the controlled device can also continuously respond to the operation instructions corresponding to the input events, thereby ensuring the cross-screen dragging control of the controlled object on the controlled device by the master device, simplifying the existing cross-screen dragging that requires multiple interactions to achieve, and also improving the input control flexibility of the master device, and also improving the operator's operating experience.

[0127] As a third optional embodiment of this embodiment, based on the above embodiment, determining that a drag operation has occurred on the target object according to the input event information can be further concretized into the following steps:

[0128] a3) According to the preset second event mapping information, the input event information is converted into parsable device event description information.

[0129] In this embodiment, the second event mapping information can be specifically understood as information for mapping the data description format involved in the interactive control of the master device with the resolvable format involved in the interactive control of the controlled device. In this embodiment, matching mapping conversion information can be found through the second event mapping information, and the input event information can be converted into resolvable device event description information through the mapping conversion information.

[0130] It can be considered that the event-related information contained in the device event description information is basically the same as the input event information, and only the expression form is different. Therefore, through the device event description information, the pressing state of the input event generated by the master device and the original input coordinates of the event can be determined. At the same time, according to the relevant mapping information of the master device side screen and the controlled device side screen in the second event mapping information, the event input screen coordinates corresponding to the original input coordinates of the event on the controlled device can be determined.

[0131] b3) parsing the device event description information, determining the operation event corresponding to the input event as a press-down move, and obtaining the event input screen coordinates involved in the press-down move.

[0132] In this embodiment, the device event description information is parsed, and through the press status and event input screen coordinates included in the device event description information, it can be determined whether the operation event corresponding to the input event is a press-to-move, and when the operation event is a press-to-move, the event input screen coordinates corresponding to each press-to-move event can be obtained.

[0133] c3) determining, according to the event input screen coordinates, that a drag operation has occurred on the target object in the first screen to the second screen.

[0134] In this embodiment, the event input screen coordinates corresponding to each press-to-move event obtained as above can be used to determine the press start position and each moving position point of the press-to-move event, so that the press-to-move trajectory, as well as the controlled object and its position can be determined through the press start position and the moving position point, and finally, it can be determined through the above information that a dragging operation has occurred on the target object in the first screen to the second screen.

[0135] Specifically, in the third optional embodiment, as an implementation method, determining that a drag operation has occurred on the target object in the first screen to the second screen according to the event input screen coordinates can be further optimized into the following steps:

[0136] c31) Determine the pressing start coordinates and the moving coordinate sequence of the pressing movement according to the event input screen coordinates.

[0137] In this embodiment, the event input screen coordinates are the coordinate information of each operation event being a press-and-move event. Through this coordinate information, it is possible to determine where the press operation was performed specifically, and which specific moving coordinate points were generated during the press-and-move process. Thus, a moving coordinate sequence can be generated based on the moving coordinate points, and the coordinates where the first press operation occurred can also be marked as the press start coordinates.

[0138] c32) Determining, based on the pressing start coordinates, whether a pressing operation has been performed on the target object in the first screen.

[0139] In this embodiment, it can be considered that the controlled device uses a coordinate system including multiple screens as the screen coordinate system, thereby determining the position area of ​​each screen in the screen coordinate system. This step can determine which screen the pressing start coordinate falls on, and which controlled object in the screen it specifically corresponds to. This step can determine the screen where the pressing start coordinate falls as the first screen, and determine the controlled object touched by the pressing start coordinate as the target object, and finally determine that the pressing operation is performed on the target object in the first screen.

[0140] c33) determining, based on the movement coordinate sequence, that a drag operation has occurred on the target object from the first screen to the second screen.

[0141] In this embodiment, the moving coordinate sequence can be considered as a coordinate sequence of moving coordinates generated by pressing and moving the target object. The moving coordinates in the moving coordinate sequence can be used to determine the pressing and moving trajectory of the target object. In this embodiment, the coordinates corresponding to pressing and lifting the target object after moving it can be determined as the end point coordinates, and the screen where the terminal coordinates are located can be determined as the second screen. Finally, through this step, it can be determined that the target object has undergone a drag operation from the first screen to the second screen.

[0142] The above technical solution of this embodiment realizes the analysis and determination of the drag operation, and the relevant information of the drag operation provides data support for the subsequent response to the drag operation

[0143] Figure 4 A schematic diagram of the structure of an interactive control device provided by an embodiment of the present disclosure is given. This embodiment is applicable to interactive control situations. The device can be implemented by software and / or hardware and can be configured on an electronic device as a master control device, which can preferably be a virtual reality device. Specifically, the electronic device as the master control device establishes a wireless connection with the controlled device, and the controlled device includes at least two screens. Specifically, the device may include: a window display module 41, an information sending module 42, and a content display module 43.

[0144] The window display module 41 is used to display at least two window interfaces in the interactive control scenario, each window interface corresponds to a screen of the controlled device, and presents the screen content of the corresponding screen;

[0145] an information sending module 42, configured to monitor an input event generated by an input device, generate input event information of the input event, and send the input event information to the controlled device, so that the controlled device determines that a drag operation has occurred on the target object based on the input event information, wherein the input event is generated in the process of an indication cursor of the input device moving from the first window interface to the second window interface;

[0146] The content display module 43 is used to receive and display the screen content fed back by the controlled device, where the screen content is a target object dragged from the first screen to the second screen, and the first screen and the second screen correspond to the first window interface and the second window interface respectively.

[0147] The interactive control device provided in this embodiment provides a complete reception implementation of obtaining the output events generated by the input device when it moves from one window interface to another window interface. Different from the prior art that can only receive input events generated in one window interface, the above scheme of this embodiment better ensures the integrity of the received input events and also ensures the simplification of operation events at the operation level. This embodiment provides basic data support for the subsequent realization of flexible dragging of controlled objects across screens.

[0148] Furthermore, the information sending module 42 may include: an event monitoring unit, which may be specifically used for:

[0149] By calling an input receiving function, monitoring the indicating cursor of the input device, the input receiving function is constructed in the operating system that the main control device relies on, and the input receiving function is constructed in association with the interactive control scene;

[0150] receiving a first input event generated by the input device when the indicating cursor is in the first window interface;

[0151] When the indicating cursor moves from the first window interface to the second window interface, the second input event generated by the input device continues to be received.

[0152] Furthermore, the input receiving function is constructed in a function library of the operating system, and a code description for establishing an association with the interactive control scenario is defined in the input receiving function;

[0153] After the wireless connection is established, the input receiving function is called and started to run through a set input calling interface.

[0154] Furthermore, the information sending module 42 may further include: an information generating unit, and the information generating unit may specifically include:

[0155] An acquisition subunit, used to acquire event attribute information corresponding to the input event, wherein the event attribute information includes: original input coordinates of the event and a pressed state;

[0156] The determination subunit is used to determine the input event information corresponding to the input event according to the preset first event mapping information and the event attribute information.

[0157] Furthermore, it is determined that the subunit can be specifically used for:

[0158] Determine the operation event represented by the input event according to the original input coordinates and the pressing state of the event;

[0159] According to the first event mapping information, event description information of the operation event is generated, and the event description information is used as the input event information.

[0160] The interactive control device provided in the embodiments of the present disclosure can execute the interactive control method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0161] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present disclosure.

[0162] Figure 5 A schematic diagram of the structure of an interactive control device provided by an embodiment of the present disclosure is given. This embodiment can be applied to interactive control situations. The device can be implemented by software and / or hardware and can be configured on an electronic device as a controlled device, which can be a mobile terminal, a PC or a server. Specifically, the controlled device can establish a wireless connection with the main control device, and the controlled device as the execution subject of this embodiment can be configured with at least two screens. Specifically, the device may include: a content sending module 51, an information receiving module 52 and an operation response module 53.

[0163] The content sending module 51 is used to send the screen content of each screen to the main control device in the interactive control scenario, so as to display at least two window interfaces on the main control device, each window interface corresponding to a screen, and present the screen content of the corresponding screen;

[0164] An information receiving module 52, used to receive input event information sent by the master control device, and determine whether a drag operation has occurred according to the input event information, wherein the input event information is generated by the master control device by monitoring input events of the input device;

[0165] The operation response module 53 is used to respond to the drag operation and feed back the screen content generated in the drag operation response to the main control device so that the main control device receives and displays the screen content, wherein the screen content is the target object dragged from the first screen to the second screen, and the first screen and the second screen correspond to the first window interface and the second window interface on the main control device respectively.

[0166] The interactive control device provided in this embodiment specifically provides a real-time response process to the press-and-move event when the controlled device side presses and moves across windows through the master device side. In the above embodiment, it is ensured that the master device will not interrupt the reception of input events due to the input device moving from one window to another window, and on the basis of being able to continuously receive input events generated across windows, it is ensured that the controlled device can also continuously respond to the operation instructions corresponding to the input events, thereby ensuring the master device's cross-screen drag control of the controlled object on the controlled device, simplifying the existing cross-screen dragging that requires multiple interactions to achieve, and also improving the input control flexibility of the master device, and also improving the operator's operating experience.

[0167] Furthermore, the information receiving module 52 may specifically include:

[0168] A mapping conversion unit, configured to convert the input event information into parsable device event description information according to preset second event mapping information;

[0169] A parsing unit, used for parsing the device event description information, determining the operation event corresponding to the input event as a press-down move, and obtaining the event input screen coordinates involved in the press-down move;

[0170] The drag determination unit is used to determine, according to the event input screen coordinates, that a drag operation has occurred on the target object in the first screen to the second screen.

[0171] Furthermore, the drag determination unit may be specifically used for:

[0172] Determine the pressing start coordinates and the moving coordinate sequence of the pressing movement according to the event input screen coordinates;

[0173] Determining, according to the pressing start coordinates, that a pressing operation has been performed on the target object in the first screen;

[0174] According to the movement coordinate sequence, it is determined that a drag operation has occurred on the target object from the first screen to the second screen.

[0175] The interactive control device provided in the embodiments of the present disclosure can execute the interactive control method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0176] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present disclosure.

[0177] Figure 6 A schematic diagram of the structure of a virtual reality device provided by an embodiment of the present disclosure is given. Figure 6 , which shows a schematic diagram of the structure of a virtual reality device 600 suitable for implementing the embodiment of the present disclosure. The virtual reality device in the embodiment of the present disclosure may include but is not limited to wearable smart devices such as head-mounted devices, virtual reality glasses, etc. Figure 6 The virtual reality device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0178] like Figure 6 As shown, the virtual reality device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. Various programs and data required for the operation of the virtual reality device 600 are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 606. An edit / output (I / O) interface 605 is also connected to the bus 606.

[0179] Typically, the following devices may be connected to the I / O interface 605: input devices 606 including, for example, a Bluetooth mouse, a handle, gestures, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 607 including, for example, a handle including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 608 including, for example, a tape, a hard disk, etc.; and communication devices 609. The communication device 609 may allow the virtual reality device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The virtual reality device 600 having various devices is shown, but it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have instead.

[0180] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0181] Figure 7A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure is given. Figure 7 , which shows an electronic device (eg, Figure 7 The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0182] like Figure 7 As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 707. An edit / output (I / O) interface 705 is also connected to the bus 707.

[0183] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0184] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 709, or installed from a storage device 708, or installed from a ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0185] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0186] The electronic device provided by the embodiment of the present disclosure and the interactive control method provided by the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0187] The embodiments of the present disclosure provide a computer storage medium on which a computer program is stored. When the program is executed by a processor, the interactive control method provided by the above embodiments is implemented.

[0188] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0189] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0190] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0191] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

[0192] A language switching operation is received, wherein the language switching operation is used to switch the current first language to a second language; while displaying the interface content in the second language, a first format of a target number in the interface content is switched to a second format corresponding to the second language.

[0193] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0194] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0195] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, constitute a limitation on the unit itself.

[0196] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0197] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0198] According to one or more embodiments of the present disclosure, [Example 1] provides an interactive control method, which is applied to a master control device, wherein the master control device establishes a wireless connection with a controlled device, and the controlled device includes at least two screens, and the method includes:

[0199] In the interactive control scenario, at least two window interfaces are displayed, each window interface corresponds to a screen of the controlled device, and the screen content of the corresponding screen is presented;

[0200] monitoring an input event generated by an input device, generating input event information of the input event, and sending the input event information to the controlled device, so that the controlled device determines that a drag operation has occurred on the target object based on the input event information, wherein the input event is generated when an indication cursor of the input device moves from the first window interface to the second window interface;

[0201] The screen content fed back by the controlled device is received and displayed, wherein the screen content is a target object dragged from a first screen to a second screen, and the first screen and the second screen correspond to the first window interface and the second window interface respectively.

[0202] According to one or more embodiments of the present disclosure, [Example 2] provides the method of Example 1, further comprising: Optionally, the monitoring of input events generated by the input device and generating input event information of the input events include:

[0203] By calling an input receiving function, monitoring the indicating cursor of the input device, the input receiving function is constructed in the operating system that the main control device relies on, and the input receiving function is constructed in association with the interactive control scene;

[0204] receiving a first input event generated by the input device when the indicating cursor is in the first window interface;

[0205] When the indicating cursor moves from the first window interface to the second window interface, the second input event generated by the input device continues to be received.

[0206] According to one or more embodiments of the present disclosure, [Example 3] provides the method of Example 1, further comprising: optionally, the input receiving function is constructed in a function library of the operating system, and a code description for establishing an association with the interactive control scenario is defined in the input receiving function;

[0207] After the wireless connection is established, the input receiving function is called and started to run through the set input calling interface.

[0208] According to one or more embodiments of the present disclosure, [Example 4] provides the method of Example 1, further comprising: Optionally, the input event information of the input event generated includes:

[0209] Acquire event attribute information corresponding to the input event, wherein the event attribute information includes: original input coordinates of the event and a pressed state;

[0210] According to the preset first event mapping information and the event attribute information, the input event information corresponding to the input event is determined.

[0211] According to one or more embodiments of the present disclosure, [Example 5] provides the method of Example 1, further comprising: Optionally, determining the input event information corresponding to the input event according to the preset first event mapping information and the event attribute information includes:

[0212] Determine the operation event represented by the input event according to the original input coordinates and the pressing state of the event;

[0213] According to the first event mapping information, event description information of the operation event is generated, and the event description information is used as the input event information.

[0214] According to one or more embodiments of the present disclosure, [Example 6] provides the method of Example 1, further comprising: optionally, the main control device is a virtual reality device;

[0215] The input device package is a handle, and the indication cursor is a handle ray;

[0216] Alternatively, the input device is a gesture, and the indicating cursor is a hand icon;

[0217] Alternatively, the input device is a Bluetooth mouse, and the indicating cursor is a mouse arrow.

[0218] According to one or more embodiments of the present disclosure, [Example 7] provides an interactive control method, which is applied to a controlled device, wherein the controlled device establishes a wireless connection with a master device, and the controlled device includes at least two screens, and the method includes:

[0219] In the interactive control scenario, the screen contents of each of the screens are sent to the main control device, so as to display at least two window interfaces on the main control device, each of which corresponds to a screen and presents the screen contents of the corresponding screen;

[0220] receiving input event information sent by the master control device, and determining, according to the input event information, that a drag operation has occurred on the target object, wherein the input event information is generated by the master control device by monitoring input events of an input device;

[0221] Respond to the drag operation, and feed back the screen content generated in the drag operation response to the main control device, so that the main control device receives and displays the screen content, wherein the screen content is the target object dragged from the first screen to the second screen, and the first screen and the second screen correspond to the first window interface and the second window interface on the main control device respectively.

[0222] According to one or more embodiments of the present disclosure, [Example 8] provides the method of Example 7, further comprising: optionally, determining that a drag operation has occurred on the target object according to the input event information, including:

[0223] According to the preset second event mapping information, the input event information is converted into parsable device event description information;

[0224] Parsing the device event description information, determining the operation event corresponding to the input event as a press-and-move, and obtaining the event input screen coordinates involved in the press-and-move;

[0225] According to the event input screen coordinates, it is determined that a drag operation has occurred on the target object in the first screen to the second screen.

[0226] According to one or more embodiments of the present disclosure, [Example 9] provides the method of Example 7, further comprising: optionally,

[0227] The step of determining, according to the event input screen coordinates, that a drag operation has occurred on the target object in the first screen to the second screen, includes:

[0228] Determine the pressing start coordinates and the moving coordinate sequence of the pressing movement according to the event input screen coordinates;

[0229] Determining, according to the pressing start coordinates, that a pressing operation has been performed on the target object in the first screen;

[0230] According to the movement coordinate sequence, it is determined that a drag operation has occurred on the target object from the first screen to the second screen.

[0231] According to one or more embodiments of the present disclosure, [Example 10] provides an interactive control device, which is configured in a master control device, wherein the master control device establishes a wireless connection with a controlled device, wherein the controlled device includes at least two screens, and the device includes:

[0232] A window display module, used to display at least two window interfaces in the interactive control scenario, each window interface corresponding to a screen of the controlled device, and present the screen content of the corresponding screen;

[0233] an information sending module, configured to monitor an input event generated by an input device, generate input event information of the input event, and send the input event information to the controlled device, so that the controlled device determines that a drag operation has occurred on the target object based on the input event information, wherein the input event is generated in the process of an indication cursor of the input device moving from the first window interface to the second window interface;

[0234] The content display module is used to receive and display the screen content fed back by the controlled device, wherein the screen content is a target object dragged from a first screen to a second screen, and the first screen and the second screen correspond to the first window interface and the second window interface respectively.

[0235] According to one or more embodiments of the present disclosure, [Example 11] provides an interactive control device, which is configured on a controlled device, wherein the controlled device establishes a wireless connection with a main control device, and the controlled device includes at least two screens, and the device includes:

[0236] A content sending module, used for sending the screen content in each of the screens to the main control device in the interactive control scenario, so as to display at least two window interfaces on the main control device, each window interface corresponding to a screen, and presenting the screen content in the corresponding screen;

[0237] An information receiving module, used to receive input event information sent by the master control device, and determine whether a drag operation has occurred according to the input event information, wherein the input event information is generated by the master control device by monitoring input events of the input device;

[0238] An operation response module is used to respond to the drag operation and feed back the screen content generated in the drag operation response to the main control device so that the main control device receives and displays the screen content, wherein the screen content is the target object dragged from the first screen to the second screen, and the first screen and the second screen correspond to the first window interface and the second window interface on the main control device respectively.

[0239] The descriptions involved in this disclosure are only preferred embodiments of the present disclosure and explanations of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in this disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned disclosure concept. For example, a technical solution formed by replacing the above-mentioned features with the technical features with similar functions disclosed in this disclosure (but not limited to).

[0240] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0241] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. An interactive control method, characterized in that: Applied to a master control device, the master control device establishes a wireless connection with a controlled device, the controlled device includes at least two screens, and the method includes: In the interactive control scenario, at least two window interfaces are displayed, each window interface corresponds to a screen of the controlled device, and the screen content of the corresponding screen is presented; monitoring an input event generated by an input device, generating input event information of the input event, and sending the input event information to the controlled device, so that the controlled device determines that a drag operation has occurred on the target object based on the input event information, wherein the input event is generated when an indication cursor of the input device moves from the first window interface to the second window interface; The screen content fed back by the controlled device is received and displayed, wherein the screen content is a target object dragged from a first screen to a second screen, and the first screen and the second screen correspond to the first window interface and the second window interface respectively.

2. The method according to claim 1, characterized in that The step of monitoring the input event generated by the input device and generating input event information of the input event includes: By calling an input receiving function, monitoring the indicating cursor of the input device, the input receiving function is constructed in the operating system that the main control device relies on, and the input receiving function is constructed in association with the interactive control scene; receiving a first input event generated by the input device when the indicating cursor is in the first window interface; When the indicating cursor moves from the first window interface to the second window interface, the second input event generated by the input device continues to be received.

3. The method according to claim 2, characterized in that The input receiving function is constructed in a function library of the operating system, and a code description for establishing an association with the interactive control scenario is defined in the input receiving function; After the wireless connection is established, the input receiving function is called and started to run through a set input calling interface.

4. The method according to claim 1, characterized in that The input event information for generating the input event includes: Acquire event attribute information corresponding to the input event, wherein the event attribute information includes: original input coordinates of the event and a pressed state; According to the preset first event mapping information and the event attribute information, the input event information corresponding to the input event is determined.

5. The method according to claim 4, characterized in that The step of determining input event information corresponding to the input event according to the preset first event mapping information and the event attribute information includes: Determine the operation event represented by the input event according to the original input coordinates and the pressing state of the event; Event description information of the operation event is generated according to the first event mapping information, and the event description information is used as the input event information.

6. The method according to any one of claims 1 to 5, characterized in that: The main control device is a virtual reality device; The input device package is a handle, and the indication cursor is a handle ray; Alternatively, the input device is a gesture, and the indicating cursor is a hand icon; Alternatively, the input device is a Bluetooth mouse, and the indicating cursor is a mouse arrow.

7. An interactive control method, characterized in that: Applied to a controlled device, the controlled device establishes a wireless connection with a master device, the controlled device includes at least two screens, and the method includes: In the interactive control scenario, the screen contents of each of the screens are sent to the main control device, so as to display at least two window interfaces on the main control device, each of which corresponds to a screen and presents the screen contents of the corresponding screen; receiving input event information sent by the master control device, and determining, according to the input event information, that a drag operation has occurred on the target object, wherein the input event information is generated by the master control device by monitoring input events of an input device; Respond to the drag operation, and feed back the screen content generated in the drag operation response to the main control device, so that the main control device receives and displays the screen content, wherein the screen content is the target object dragged from the first screen to the second screen, and the first screen and the second screen correspond to the first window interface and the second window interface on the main control device respectively.

8. The method according to claim 7, characterized in that The determining, according to the input event information, that a drag operation has occurred on the target object includes: According to the preset second event mapping information, the input event information is converted into parsable device event description information; Parsing the device event description information, determining the operation event corresponding to the input event as a press-and-move, and obtaining the event input screen coordinates involved in the press-and-move; According to the event input screen coordinates, it is determined that a drag operation has occurred on the target object in the first screen to the second screen.

9. The method according to claim 8, characterized in that The step of determining, according to the event input screen coordinates, that a drag operation has occurred on the target object in the first screen to the second screen, includes: Determine the pressing start coordinates and the moving coordinate sequence of the pressing movement according to the event input screen coordinates; Determining, according to the pressing start coordinates, that a pressing operation has been performed on the target object in the first screen; According to the movement coordinate sequence, it is determined that a drag operation has occurred on the target object from the first screen to the second screen.

10. An interactive control device, characterized in that: The apparatus is configured on a master control device, the master control device establishes a wireless connection with a controlled device, the controlled device includes at least two screens, and the apparatus includes: A window display module, used to display at least two window interfaces in the interactive control scenario, each window interface corresponding to a screen of the controlled device, and present the screen content of the corresponding screen; an information sending module, configured to monitor an input event generated by an input device, generate input event information of the input event, and send the input event information to the controlled device, so that the controlled device determines that a drag operation has occurred on the target object based on the input event information, wherein the input event is generated in the process of an indication cursor of the input device moving from the first window interface to the second window interface; The content display module is used to receive and display the screen content fed back by the controlled device, wherein the screen content is a target object dragged from a first screen to a second screen, and the first screen and the second screen correspond to the first window interface and the second window interface respectively.

11. An interactive control device, characterized in that: The device is configured on a controlled device, the controlled device establishes a wireless connection with a main control device, the controlled device includes at least two screens, and the device includes: A content sending module, used for sending the screen content in each of the screens to the main control device in the interactive control scenario, so as to display at least two window interfaces on the main control device, each window interface corresponding to a screen, and presenting the screen content in the corresponding screen; An information receiving module, used to receive input event information sent by the master control device, and determine whether a drag operation has occurred according to the input event information, wherein the input event information is generated by the master control device by monitoring input events of the input device; An operation response module is used to respond to the drag operation and feed back the screen content generated in the drag operation response to the main control device so that the main control device receives and displays the screen content, wherein the screen content is the target object dragged from the first screen to the second screen, and the first screen and the second screen correspond to the first window interface and the second window interface on the main control device respectively.

12. A virtual reality device, characterized in that: include: one or more input devices; one or more processors; a storage device for storing one or more programs, The one or more programs are executed by the one or more processors, so that the one or more processors implement the interactive control method according to any one of claims 1 to 6.

13. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, The one or more programs are executed by the one or more processors, so that the one or more processors implement the interactive control method according to any one of claims 1 to 9.

14. A storage medium containing computer executable instructions, characterized in that: The computer executable instructions are used to perform the interactive control method as described in any one of claims 1-9 when executed by a computer processor.

15. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the interactive control method according to any one of claims 1 to 9.

Images